Picture by Writer
# Introduction
Though in trendy information science you’ll primarily discover Jupyter notebooks, Pandas, and graphical dashboards, they don’t at all times provide the degree of management you would possibly want. Then again, command-line instruments is probably not as intuitive as you want, however they’re highly effective, light-weight, and far sooner at executing the precise jobs they’re designed for.
For this text, I’ve tried to create a steadiness between utility, maturity, and energy. You’ll discover some classics which might be practically unavoidable, together with extra trendy additions that fill gaps or optimize efficiency. You’ll be able to even name this a 2025 model of a must have CLI instruments checklist. For individuals who aren’t accustomed to CLI instruments however need to be taught, I’ve included a bonus part with assets within the conclusion, so scroll all the way in which down earlier than you begin together with these instruments in your workflow.
# 1. curl
curl is my go-to for making HTTP requests like GET, POST, or PUT; downloading information; and sending/receiving information over protocols akin to HTTP or FTP. It’s supreme for retrieving information from APIs or downloading datasets, and you’ll simply combine it with data-ingestion pipelines to drag JSON, CSV, or different payloads. One of the best factor about curl is that it’s pre-installed on most Unix methods, so you can begin utilizing it straight away. Nonetheless, its syntax (particularly round headers, physique payloads, and authentication) might be verbose and error-prone. When you’re interacting with extra complicated APIs, you could want an easier-to-use wrapper or Python library, however figuring out curl continues to be a vital plus for fast testing and debugging.
# 2. jq
jq is a light-weight JSON processor that allows you to question, filter, rework, and pretty-print JSON information. With JSON being a dominant format for APIs, logs, and information interchange, jq is indispensable for extracting and reshaping JSON in pipelines. It acts like “Pandas for JSON within the shell.” The most important benefit is that it gives a concise language for coping with complicated JSON, however studying its syntax can take time, and very massive JSON information might require extra care with reminiscence administration.
# 3. csvkit
csvkit is a set of CSV-centric command-line utilities for reworking, filtering, aggregating, becoming a member of, and exploring CSV information. You’ll be able to choose and reorder columns, subset rows, mix a number of information, convert from one format to a different, and even run SQL-like queries towards CSV information. csvkit understands CSV quoting semantics and headers, making it safer than generic text-processing utilities for this format. Being Python-based means efficiency can lag on very massive datasets, and a few complicated queries could also be simpler in Pandas or SQL. In the event you want velocity and environment friendly reminiscence utilization, take into account the csvtk toolkit.
# 4. qwk / sed
Hyperlink (sed): https://www.gnu.org/software program/sed/guide/sed.html
Traditional Unix instruments like awk and sed stay irreplaceable for textual content manipulation. awk is highly effective for sample scanning, field-based transformations, and fast aggregations, whereas sed excels at textual content substitutions, deletions, and transformations. These instruments are quick and light-weight, making them good for fast pipeline work. Nonetheless, their syntax might be non-intuitive. As logic grows, readability suffers, and you could migrate to a scripting language. Additionally, for nested or hierarchical information (e.g., nested JSON), these instruments have restricted expressiveness.
# 5. parallel
GNU parallel quickens workflows by operating a number of processes in parallel. Many information duties are “mappable” throughout chunks of information. Let’s say you need to execute the identical transformation on lots of of information—parallel can unfold work throughout CPU cores, velocity up processing, and handle job management. You need to, nevertheless, be conscious of I/O bottlenecks and system load, and quoting/escaping might be tough in complicated pipelines. For cluster-scale or distributed workloads, take into account resource-aware schedulers (e.g., Spark, Dask, Kubernetes).
# 6. ripgrep (rg)
ripgrep (rg) is a quick recursive search instrument designed for velocity and effectivity. It respects .gitignore by default and ignores hidden or binary information, making it considerably sooner than conventional grep. It’s good for fast searches throughout codebases, log directories, or config information. As a result of it defaults to ignoring sure paths, you could want to regulate flags to look all the things, and it isn’t at all times out there by default on each platform.
# 7. datamash
datamash gives numeric, textual, and statistical operations (sum, imply, median, group-by, and many others.) instantly within the shell through stdin or information. It’s light-weight and helpful for fast aggregations with out launching a heavier instrument like Python or R, which makes it supreme for shell-based ETL or exploratory evaluation. Nevertheless it’s not designed for very massive datasets or complicated analytics, the place specialised instruments carry out higher. Additionally, grouping very excessive cardinalities might require substantial reminiscence.
# 8. htop
htop is an interactive system monitor and course of viewer that gives dwell insights into CPU, reminiscence, and I/O utilization per course of. When operating heavy pipelines or mannequin coaching, htop is extraordinarily helpful for monitoring useful resource consumption and figuring out bottlenecks. It’s extra user-friendly than conventional high, however being interactive means it doesn’t match properly into automated scripts. It might even be lacking on minimal server setups, and it doesn’t exchange specialised efficiency instruments (profilers, metrics dashboards).
# 9. git
git is a distributed model management system important for monitoring adjustments to code, scripts, and small information property. For reproducibility, collaboration, branching experiments, and rollback, git is the usual. It integrates with deployment pipelines, CI/CD instruments, and notebooks. Its disadvantage is that it’s not meant for versioning massive binary information, for which Git LFS, DVC, or specialised methods are higher suited. The branching and merging workflow additionally comes with a studying curve.
# 10. tmux / display screen
Terminal multiplexers like tmux and display screen allow you to run a number of terminal periods in a single window, detach and reattach periods, and resume work after an SSH disconnect. They’re important if you could run lengthy experiments or pipelines remotely. Whereas tmux is beneficial on account of its lively growth and suppleness, its config and keybindings might be tough for newcomers, and minimal environments might not have it put in by default.
# Wrapping Up
In the event you’re getting began, I’d suggest mastering the “core 4”: curl, jq, awk/sed, and git. These are used in every single place. Over time, you’ll uncover domain-specific CLIs like SQL purchasers, the DuckDB CLI, or Datasette to fit into your workflow. For additional studying, try the next assets:
- Knowledge Science on the Command Line by Jeroen Janssens
- The Artwork of Command Line on GitHub
- Mark Pearl’s Bash Cheatsheet
- Communities just like the unix & command-line subreddits usually floor helpful methods and new instruments that can increase your toolbox over time.
Kanwal Mehreen is a machine studying engineer and a technical author with a profound ardour for information science and the intersection of AI with drugs. She co-authored the book “Maximizing Productiveness with ChatGPT”. As a Google Technology Scholar 2022 for APAC, she champions range and educational excellence. She’s additionally acknowledged as a Teradata Variety in Tech Scholar, Mitacs Globalink Analysis Scholar, and Harvard WeCode Scholar. Kanwal is an ardent advocate for change, having based FEMCodes to empower girls in STEM fields.
