The curl command-line tool is one of the most useful and versatile programs you can learn. Its versatility and comprehensive HTTP implementation mean that, if there’s a URL for it, curl can do it.

8
Fetch a Web Page

In its most basic form, curl fetches a URL and displays its contents on your terminal:

curl info.cern.ch

What you see will depend on the content type of the URL. If you request a web page, you’ll see its HTML source:

You can redirect this output to a file to save a web page:

curl info.cern.ch > info.cern.ch.html

Notice how curl detects that you’re redirecting the normal output and sends timing information to stderr:

A URL may contain characters with special meaning to the shell, like “?” or “#.” You’ll need to be careful to avoid problems this may cause, so make sure you surround URLs with single quotes if they contain special characters. If you’re not sure, be cautious and always quote them:

curl 'info.cern.ch'

7
Download a File

If you try to use curl to download a binary file—like an image or a Microsoft Word document—you’ll see an error telling you that “Binary output can mess up your terminal.” This is because curl will print the response body that it receives to output. In the case of an image, this is almost certainly not what you want to be doing.

The error message mentions the –output option, which lets you specify a file to save the output to rather than print it. You can also use -o as a shorter equivalent:

curl -o neo-the-cat.jpg https://placecats.com/neo/300/200

This will save the file at that URL in the current directory. As before, since curl is no longer outputting the response, it provides a lot of detail about the time your download takes. If you want a simple progress bar instead, use the -# or –progress-bar option:

curl --progress-bar -o neo-the-cat.jpg https://placecats.com/neo/300/200

6
Install Software

You may sometimes come across installation instructions that ask you to use curl. For example, atuin—a replacement for the history command—has the following installation instruction:

curl --proto '=https' --tlsv1.2 -LsSf https://setup.atuin.sh | sh

If you open that URL in a web browser, you’ll see a shell script that performs installation and setup. You can download it and run it manually if you want, the shell one-liner is just a quicker way. Note that the output—the contents of this script—is piped to sh which will run it as a shell script.

The suggested curl usage in the atuin example uses these options:

• –proto ‘=https’ to ensure curl uses the HTTPS protocol.
• –tlsv1.2 to force curl to use TLS version 1.2 or greater.
• -L to automatically follow redirects.
• -sS to suppress the progress meter but still show error messages.
• -f to treat an HTTP error as a command error. This is to ensure that no error message gets piped to sh, avoiding a potentially dangerous action.

5
Check Your IP Address

This practical use for curl is a quick and easy way of discovering your public IP address, thanks to the service at ifconfig.co:

curl ifconfig.co

The site responds with your public IP address:

This default output is plain text which makes it an excellent resource for automation. You might notice that you get a very different result if you go to the same URL in your web browser:

This is because the site behaves differently based on your user agent. By default, curl sends something like “curl/8.7.1” in the User-Agent header. If you ever need to set this to something else, e.g. for testing purposes, use the -A/–user-agent option:

curl -s --user-agent "my pretend browser" ifconfig.co | head

Now the ifconfig site will respond with HTML, just as it does in a web browser:

4
Show Response or Request Headers

Sometimes, you want to fetch just the response headers from a URL. You might be debugging an error or checking cache headers to see if you need to redownload a file. To view just the response headers, use -I/–head:

curl -I example.com

This will send a HEAD request, which not all web servers support. If you get an error using this method, you can tell curl to send a GET request and only output the headers:

curl -I -X GET example.com

If you want to see the request headers that curl sends, use the –verbose/-v option:

3
Access an API

If you’re working with an API, a good client like Postman or Posting will help you automate and debug the task. But curl can be just as useful for exploring how an API works.

The DummyJSON service is a good place to start. This site provides an example API with dummy data that you can use to test client apps and your own programs. You can use different endpoints to check redirects, test authorization, and simulate time delays or random responses.

Trying out the service with curl is as simple as passing the appropriate endpoint:

curl https://dummyjson.com/test

You should see a simple JSON document in response, with two properties. The method property will reflect the HTTP method of your request:

Once you’ve fetched data from an API, you can easily process it using standard command-line tools. For example, you can process JSON data using jq. This pipeline fetches product data using curl, then passes it to jq to extract the first product:

curl https://dummyjson.com/products | jq '.products[0]' | more

2
Fetch an RSS Feed

One of my current projects is a shell-based RSS client. At its core lies curl because the tool is so convenient and supports every low-level HTTP operation I need. For example, you can fetch an RSS feed and examine its contents using curl:

curl -s https://www.w3.org/blog/feed/ | more

If you have libxml installed, you can process the XML output using xmllint:

curl -s https://www.w3.org/blog/feed/ | xmllint --xpath '//channel/title/text()' -

This pipeline uses xmllint to evaluate an XPath expression that extracts data from XML, in this case, the title of the channel:

1
Use curl Programmatically

The cURL project is about more than just curl, the command-line tool. It also includes a library, libcurl, that curl uses itself. This library is highly portable and available for all manner of operating systems including Linux, Windows, macOS, and Android. libcurl comes with a C API, but there are bindings for many other languages like C++, Python, Rust, and Go.

The library mirrors much of what the command-line tool does, particularly its options and their behavior. Here’s a basic example of PHP usage:

<?php
$ch = curl_init("http://www.example.com/");
$fp = fopen("www-example-com.html", "w");

curl_setopt($ch, CURLOPT_FILE, $fp);
curl_setopt($ch, CURLOPT_HEADER, 0);

curl_exec($ch);

if (curl_error($ch))
    fwrite($fp, curl_error($ch));

curl_close($ch);
fclose($fp);
?>

The curl_init() function starts a session and returns a handle that you can use to access it. You can control the library’s behavior using the options that curl supports via curl_setopt(). curl_exec() actually sends the request while curl_error() and curl_close() are useful admin functions.

This example uses the CURLOPT_FILE setting to write the response body to an open file handle. Note that this is the equivalent of curl’s –output option.

If you’re looking for other terminal-based tools to master, learn some basic git commands or look at some bash script examples.