bash curl - ghdrako/doc_snipets GitHub Wiki

curl converter to programing language

libcurl

https://everything.curl.dev/libcurl/libcurl

To compile

    curl https://ifconfig.me --libcurl ip_fetcher.c 
    # Output: your ip address, and a file ip_fetcher.c

    gcc -o ip_fetcher ip_fetcher.c -lcurl
    # Output: no errors, just a file ip_fetcher

    ./ip_fetcher
    # Output: your ip address

It might be even easier this way:

    $ curl https://ifconfig.me --libcurl ip_fetcher.c
    $ make LDFLAGS=-lcurl ip_fetcher
    $ ./ip_fetcher

Curl

https://myxml.in/curl-command-builder.html # best curl builder
https://curl.se/docs/manual.html
https://curlbuilder.com/ # construct curl command line
https://everything.curl.dev/ # book
https://reqbin.com/curl # run curl online with examples
https://curlconverter.com/ # convert curl to programming language ex python
https://github.com/alexkart/curl-builder
https://curl.se/docs/httpscripting.html
https://docs.gitlab.com/ee/development/documentation/restful_api_styleguide.html
https://docs.gitlab.com/ee/api/branches.html

Mutual TLS

$ curl --cert client.crt --key client.key --cacert ca.crt https://myserver.internal.net:443

Measure latency

https://sylwit.medium.com/how-to-measure-latency-with-curl-2d3bfc312601


$ curl -o /dev/null -s -w {time_connect}:%{time_starttransfer}:%{time_total}\\n  http://www.google.com
$ curl -vw "\ndnslookup: %{time_namelookup} | connect: %{time_connect} | appconnect: %{time_appconnect} | pretransfer: %{time_pretransfer} | redirect: %{time_redirect} | starttransfer: %{time_starttransfer} | total: %{time_total} | size: %{size_download}\n" -so /dev/null http://10.222.199.2:5432

-w - Write out

curl_format.txt:
\n   
   namelookup: %{time_namelookup}\n
      connect: %{time_connect}\n
   appconnect: %{time_appconnect}\n
  pretransfer: %{time_pretransfer}\n
     redirect: %{time_redirect}\n
starttransfer: %{time_starttransfer}\n
 — — — — — \n
        total: %{time_total}\n
         size: %{size_download}\n

curl -w "@curl_format.txt" -so /dev/null https://kubernetes.io/

curl has many different supported protocols. However, curl will use HTTP protocol by default if no protocol is provided.

curl example.com

You can call a specific protocol by prefacing the URL with the protocol name.

curl http://example.com
curl ftp://example.com

list of curl supported protocols:

DICT
FILE
FTP
FTPS
GOPHER
HTTP
HTTPS
IMAP
IMAPS
LDAP
POP3
RTMP
RTSP
SCP
SFTP
SMB
SMBS
TELNET
TFTP

Wymuszenie tlsv1.3

curl -TLSv1.3 url

Insecure mode

curl -k url
curl --insecure url
curl --insecure [options] url
curl --insecure -I url

curl -k  is giving you insecure https connection to the server. 
curl -k https://example.com:8443/cli/agentCLI -u username:password

To download the .pem from the server: * that is usually found in /etc/ssl/ if you have access to the server, * or that you can download, using:

echo "HEAD / HTTP/1.0\n Host: example.com\n\n EOT\n" | 
openssl s_client -prexit -connect example.com:8443 > cert.pem

to your computer, keep only the part between BEGIN CERTIFICATE and END CERTIFICATE within the file (including the BEGIN/END lines) and give it as parameter to the --cacert option, you might also download it. Then you'll get to authenticate your server each time you connect!

curl --cacert cert.pem https://example.com:8443/cli/agentCLI -u username:password

Verbose mode

curl -v url
curl -v http://example.com -o saved
curl -v telnet://<ip>:22

curl https://mysite.com --cacert gd_bundle.crt

ftp

curl ftp://user:[email protected]/
curl "ftp://example.com/foo;type=A" # ascii
curl "ftp://example.com/foo;type=I" # binary
curl "ftp://example.com/foo;type=D" # resource is directory

ipv6

curl http://[2a04:4e42::561]/
curl http://[fdea::1]:8080/

url globing -O option

curl -O "http://example.com/[1-100].png"
curl -O "http://example.com/[001-100].png"
curl -O "http://example.com/[0-100:2].png"
curl -O "http://example.com/section[a-z].html"  # alfabetical
curl -O "http://example.com/{one,two,three,alpha,beta}.html" # list
curl -O "http://example.com/{Ben,Alice,Frank}-{100x100,1000x1000}.jpg" # combination
curl -O "http://example.com/{web,mail}-log[0-6].txt"
curl "http://{one,two}.example.com" -o "file_#1.txt"  # output reference to glob #[num]
curl "http://{site,host}.host[1-5].example.com" -o "subdir/#1_#2"

Config file -K/--config option

curl -K cmdline.txt http://example.com
cmdline.txt:

--location
# ask to do HEAD request
--head
user-agent "Everything-is-an-agent"
url = "http://example.com"

Default config file

$CURL_HOME/.curlrc
$XDG_CONFIG_HOME/.curlrc
$HOME/.curlrc
Windows: %USERPROFILE%\\.curlrc
Windows: %APPDATA%\\.curlrc
Windows: %USERPROFILE%\\Application Data\\.curlrc

Password

curl -u alice:12345 http://example.com/

Progress

-s / --silent
-S / --show-error
-# / --progress-bar

Trace options --trace [filename]

curl --trace dump http://example.com
curl -v --trace-time http://example.com

Write out --write-out or -w

It offers a large range of variables that you can include in the output, variables that have been set with values and information from the transfer.

curl -w "formatted string" http://example.com/
curl -w @filename http://example.com/ # read from file
curl -w @- http://example.com/ # read from stdin
curl -w "Type: %{content_type}\nCode: %{response_code}\n" http://example.com # variables that are available are accessed by writing %{variable_name} in the                                                  
string and that variable will then be substituted by the correct value

Storing downloads -o

curl -o output.html http://example.com/
curl -o /tmp/index.html http://example.com/
curl http://example.com -o ../../folder/savethis.htm
curl -o file.txt ftp://example.com/path/to/file-name.ext
curl -o file.html http://example.com/file.html
curl -O http://example.com/file.html # (-O / --remote-name) selects the local file name to use by picking the file name part of the URL

--remote-name-all # makes -O the default operation for all given URLs

Compression - ask HTTP and HTTPS servers to provide compressed versions of the data and then perform automatic decompression of it on arrival

curl --compressed http://example.com/

Multiple downloads

curl -o one.html http://example.com/1 http://example.com/2
curl -O -O http://example.com/1 http://example.com/2  # 2 files so two -0 or curl --remote-name-all http://example.com/1 http://example.com/2

Limits

curl https://example.com/ --limit-rate 200K # Rate limiting
curl --max-filesize 100000 https://example.com/ # Maximum file size

Storing metadata in file system

--xattr

RAW - disables all internal HTTP decoding of content or transfer encodings and instead makes curl passed on unaltered, raw, data

--raw

Retry

curl --retry 5 --retry-max-time 120 https://example.com
curl --retry 12 --retry-all-errors https://example.com

Upload

# method POST -d or --data
# method PUT
curl -T uploadthis http://example.com/
# FTP
curl -T uploadthis ftp://example.com/this/directory/
curl -T uploadthis ftp://example.com/this/directory/remotename # select a different file name on the remote side than what you have
# SMTP
curl -T mail smtp://mail.example.com/ --mail-from [email protected]

Local port number

The local port number is usually randomly assigned to your TCP connection by the network stack and you normally do not have to think about it much further. You can specify which local ports curl should bind the connection to. You can specify a single port number to use, or a range of ports.

curl --local-port 4000-4200 https://example.com/

Keep alive

curl --no-keepalive https://example.com/
url --keepalive-time 300 https://example.com/  # in seconds

Enable TLS

curl --ssl-reqd ftp://ftp.example.com/file.txt
curl --ssl ftp://ftp.example.com/file.txt
curl https://www.example.com/

CA store

curl needs a "CA store", a collection of CA certificates, to verify the TLS server it talks to.

You can point out a specific CA bundle to use in the TLS handshake with the --cacert command line option. That bundle needs to be in PEM format. You can also set the environment variable CURL_CA_BUNDLE to the full path.

CA store on Windows - search file curl-ca-bundle.crt in application's directory/current working directory/Windows System directory/Windows Directory

Client certificates

TLS client certificates are a way for clients to cryptographically prove to servers that they are truly the right peer (also sometimes known as Mutual TLS or mTLS).

curl offers options to let you specify a single file that is both the client certificate and the private key concatenated using --cert, or you can specify the key file independently with --key:

curl --cert mycert:mypassword https://example.com
curl --cert mycert:mypassword --key mykey https://example.com
curl --cert mycert:mypassword --cert-type PEM \
     --key mykey --key-type PEM https://example.com

curl -H "Host: webapp.istioinaction.io" \
     --cacert ch4/certs/2_intermediate/certs/ca-chain.cert.pem \
     --resolve webapp.istioinaction.io:443:127.0.0.1 \
     --cert ch4/certs/4_client/certs/webapp.istioinaction.io.cert.pem \
     --key ch4/certs/4_client/private/webapp.istioinaction.io.key.pem \
     https://webapp.istioinaction.io:443/api/catalog \

Conditionals

# Check by modification date
curl -z "Jan 10, 2017" https://example.com/file -O  # Download the file only if it is newer than a specific date -z / --time-cond
curl --time-cond "-Jan 10, 2017" https://example.com/file -O  # file only if it is older than the specific time by prefixing the date with a dash
curl --time-cond "Sun, 12 Sep 2004 15:05:58 -0700" https://www.example.org/file.html
curl -z file.html https://example.com/file.html -O # -z option can also extract and use the timestamp from a local file, which is handy to only download a file if it has been updated remotely
curl -z file.html -o file.html --remote-time https://example.com/file.html # combine the use of -z with the --remote-time flag, which sets the time of the locally created file to the same timestamp as the remote file had
# Check by modification of content
curl --etag-save etags.txt https://example.com/file -o output # download a remote file and save its ETag (if it provides any) in a separate "cache" by using the --etag-save
curl --etag-compare etag.txt https://example.com/file -o output # use that previously saved etag and make sure to only download the file again if it has changed
curl --etag-compare etag.txt --etag-save etag.txt https://example.com/file -o output # combination both option

Content-Type

curl -d '{json}' -H 'Content-Type: application/json' https://example.com
curl --data-binary @filename http://example.com/
curl --json '{"tool": "curl"}' https://example.com/ # Send a basic JSON object to a server
curl --json @json.txt https://example.com/
echo '{"a":"b"}' | curl --json @- https://example.com/
curl --json @json.txt --json ", "end": "true"}' https://example.com/
jo -p name=jo n=17 parser=false | curl --json @- https://example.com/  # jo tool to create json object
curl --json '{"tool": "curl"}' https://example.com/ | jq  # send json and pretty print response
jo -p name=jo n=17 | curl --json @- https://example.com/ | jq

Conver to GET

-G or --get option, which takes all data you have specified with the different -d variants and appends that data to the inputted URL e.g. http://example.com separated with a '?' and then makes curl send a GET instead

curl -G --data-urlencode "name=daniel stenberg" https://example.com/

Converting a web form

curl -F "person=Mr Smith" -F [email protected] -F "username=bob123" \
https://example.com/user/submit.cgi

-d vs -F - regular POST or multipart post

HTTP redirects

It is the server sending back an instruction to the client instead of giving back the contents the client wanted. The server says “go look over here instead for that thing you asked for“. All redirects also need to send back a Location: header with the new URI to ask for, which can be absolute or relative.

curl's tradition of only doing the basics unless you tell it differently, it does not follow HTTP redirects by default. Use the -L, --location option to tell it to do that.

Modify the HTTP request

Each HTTP transfer starts with curl sending an HTTP request. That request consists of a request line and a number of request headers

curl http://example.com/file # simple GET request with ```GET /file HTTP/1.1```
curl http://example.com/file -X DELETE # -X or --request allow change method

Customize headers

curl -H "Host: test.example" http://example.com/
curl -H "Elevator: floor-9" http://example.com/
curl -H "User-Agent:" http://example.com/ # switch off User-Agent header
curl -H "Empty;" http://example.com # add header with no content
curl -H "Host: webapp.istioinaction.io" https://localhost/api/catalog
curl -H "Host: catalog.istioinaction.io" -H "x-istio-cohort: internal" http://localhost/items

User-agent

The User-Agent is a header that each client can set in the request to inform the server which user-agent it is.

The default header value is 'curl/[version]', as in User-Agent: curl/7.54.1 for curl version 7.54.1.

You can set any value you like, using the option -A or --user-agent plus the string to use or, as it's just a header, -H "User-Agent: foobar/2000"

PUT Method

curl -T localfile http://example.com/new/resource/file
curl -d "data to PUT" -X PUT http://example.com/new/resource/file

HTTP/2 One of the primary features in the HTTP/2 protocol is the ability to multiplex several logical stream over the same physical connection.

curl --http2 http://example.com

Mesure time request

time curl -s -o /dev/null -H \
"Host: simple-web.istioinaction.io" localhost

Get the HTTP Header

Using –head Option

$ curl --head --silent https://www.example.com

Get the HTTP Status

$ curl --silent --head https://www.example.com | awk '/^HTTP/{print $2}' # show response status code
$ curl --silent --output /dev/null --write-out "%{http_code}" --location 'https://mail.google.com' # the same more readably

Certificate Validation Methods

-k/--insecure - curl not validate the peer
--cacert [file] - to verify the remote server when connecting, we can point out the CA cert
--with-ca-bundle=FILE - add the CA cert for our server to the existing default CA certificate store
--with-ca-path=PATH
-v or –verbose to display the raw messages that we send to or receive from the opposite end
-L or –location parameter will make curl automatically follow the redirect(s).
-A or –user-agent parameter means user agent. Some web servers require it, some don’t.

$ openssl s_client -connect icanhazip.com:443 -showcerts > icanhazip_com.pem
$ curl --verbose --cacert icanhazip_com.pem https://icanhazip.com

Displaying Message Header Only

curl -vsA "Mozilla" -o /dev/null https://www.baeldung.com 2>&1 | grep '>\|<'

curl puts a ‘>’ character preceding the request message, and ‘<‘ preceding the response message pattern ‘>|<‘ in grep - we want to get the output from the curl command that contains only a ‘>’ or ‘<‘ character

$ curl -vsA "Mozilla" -o /dev/null https://www.baeldung.com 2>&1 | sed '/^[* {}]/d'

/^[* {}]/d: remove line(s) starting with ‘*’, space, ‘{‘, or ‘}’ character

Displaying Request Message Header Only

$ curl -vs https://www.baeldung.com 2>&1 >/dev/null | grep '>'
$ curl -vso /dev/null https://www.baeldung.com 2>&1 | sed '/^[* {}<]/d'

$ curl -vs https://www.baeldung.com 2>&1 >/dev/null | grep '>' | cut -c1-2 --complement
$ curl -vso /dev/null https://www.baeldung.com 2>&1 | sed '/^[* {}<]/d; s/> //;'

Paginstion

The link header provides the URL for the previous, next, first, and last page of results:

The URL for the previous page is followed by rel="prev".
The URL for the next page is followed by rel="next".
The URL for the last page is followed by rel="last".
The URL for the first page is followed by rel="first". In some cases, only a subset of these links are available.

$ curl --include --header --request GET --url "https://api.github.com/repos/octocat/Spoon-Knife/issues" --head --silent "Accept: application/vnd.github+json"|grep link
link: <https://api.github.com/repositories/1300192/issues?page=2>; rel="next", <https://api.github.com/repositories/1300192/issues?page=527>; rel="last"

pageoffset=50
# Get result of page 1 to count for paging calls
result1=$(curl "https://entreprise.data.gouv.fr/api/sirene/v1/full_text/MONTPELLIERAIN?per_page="$pageoffset"&page=1")

# Prepare to be able to get/calculate pages and hence number of API calls and variables to use when looping through each page
tot_result=$(echo $result1|jq -r .total_results)
tot_page=$(echo $result1|jq -r .total_pages) # Here tot page is available. You may need to calculate if you only get total result
calculated_tot_page=$(echo "if ( $tot_result%$pageoffset ) $tot_result/$pageoffset+1 else $tot_result/$pageoffset" |bc)

# Take each page, get it and save as a separate file
for ((i=1;i<=tot_page;i++));
    do curl -s "https://entreprise.data.gouv.fr/api/sirene/v1/full_text/MONTPELLIERAIN?per_page="$pageoffset"&page="$i | jq '.[0].etablissement[]' --slurp >| "/tmp/content"$i".json";
       sleep 0.3; # Add delay of 0.3s to not be kicked due to rate limitations
done;

# Merge your JSONs "et voilà!" as we say in French
cat /tmp/content*.json | jq -s . >| /tmp/out.json

curl -I "https://api.github.com/search/code?q=addClass+user:mozilla" The -I

curl -O 'https://api.mysite.com/info?page=[1-100]'

for ((i=0; ; i+=100)); do
    contents=$(curl -u "username:password" -H "Content-Type: application/json" "https://<url>/api/core/v3/places?count=100&startIndex=$i")
    echo "$contents" > $i.json
    if jq -e '.list | length == 0' >/dev/null; then 
       break
    fi <<< "$contents"
done

Get the total entries from the first page:

total=$(curl "$url&page=1" | jq .total_entries)

GitLab branches

Use https://gitlab.example.com/api/v4/ as an endpoint.
Wherever needed use this personal access token: <your_access_token>.
Always put the request first. GET is the default so you don’t have to include it.
Wrap the URL in double quotes (").
Prefer to use examples using the personal access token and don’t pass data of username and password. Get the details of a group:

curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/gitlab-org"

Create a new project under the authenticated user’s namespace:

curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects?name=foo"

creates a new group. Be aware of the use of single (') and double (") quotes.

curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --header "Content-Type: application/json" \
     --data '{"path": "my-group", "name": "My group"}' "https://gitlab.example.com/api/v4/groups"

List repository branches

curl --header “PRIVATE-TOKEN: <your_access_token>” https://gitlab.example.com/api/v4/projects/5/repository/branches

Get single repository branch

curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/repository/branches/main"

Create a new branch in the repository

curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/repository/branches?branch=newbranch&ref=main"

Delete repository branch

curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/repository/branches/newbranch"

Delete merged branches

curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/repository/merged_branches"

Curl variables

https://daniel.haxx.se/blog/2023/07/31/introducing-curl-command-line-variables/