Repository navigation
Bug: instagram title #1670
Description
Provide a screenshot and describe the bug
When submitting an instagram URL into archivebox, the fetched title is only "instagram".
I noticed the title is set in javascript in some circumstances but not all:
- if a firefox/chrome user-agent is sent, the real title is set with javascript
- with a curl user-agent, a full title is returned
For example, I tried this outside archivebox:curl https://www.instagram.com/instagram/reel/DHd6PNKs_Mj/ | htmlq title
And it printed
<title>Instagram | Nailed it 😆🛹👏
#InTheMoment
Video by @jonjustino
Music by The Dave Brubeck Quartet | Instagram</title>
So I tried to set CURL_USER_AGENT=curl/8.13.0-rc3 env var in archivebox and submitted another instagram URL, but archivebox still only extracted title "instagram"
Steps to reproduce
1. submit an instagram URL (like `https://www.instagram.com/instagram/reel/DHd6PNKs_Mj/`) with at least `title` extractorLogs or errors
ArchiveBox Version
v0.7.3How did you install the version of ArchiveBox you are using?
Docker (or Podman/LXC/K8s/TrueNAS/Proxmox/etc)
What operating system are you running on?
Linux (Ubuntu/Debian/Arch/Alpine/etc.)
What type of drive are you using to store your ArchiveBox data?
- some of
data/is on a local SSD or NVMe drive - some of
data/is on a spinning hard drive or external USB drive - some of
data/is on a network mount (e.g. NFS/SMB/Ceph/GlusterFS/etc.) - some of
data/is on a FUSE mount (e.g. SSHFS/RClone/S3/B2/Google Drive/Dropbox/etc.)
Activity
hydrargyrum commentedon Mar 29, 2025
with wget:
wget --header "User-Agent: curl" -O- https://www.instagram.com/instagram/reel/DHd6PNKs_Mj/ | htmlq titlewget -O- https://www.instagram.com/instagram/reel/DHd6PNKs_Mj/ | htmlq title, yielding<title id="pageTitle">Update your browser</title>hydrargyrum commentedon Mar 29, 2025
if i'm reading the code correctly:
cmdisn't used anywhere in the function! (except for the result reporting, which is a big lie)i would have no idea how to fix this, it's quite a mess
pirate commentedon Mar 30, 2025
Trust me it's that way for good reason, the cmd is assembled so the user has something to copy-paste to debug further, this is the case for several of the Python based extractors. Curl wont exactly replicate what archivebox does but it's close enough that it will help the user find the source of the issue 99% of the time.
hydrargyrum commentedon Mar 30, 2025
sure but in the end:
all of this is very confusing for the user, and i can't see a workaround to make archivebox extract the title correctly
pirate commentedon Mar 30, 2025
If you'd like to submit a PR that applies
CURL_USER_AGENTcorrectly to the title curl method I'd happily review it.PRs to improve the docs are also welcome: https://github.com/ArchiveBox/docs
pirate commentedon Apr 1, 2025
oh wait you were on
v0.7.3! no wonder. the newer betas already added the configurable user agent when downloading the url in python. it doens't make sense to use the CURL_USER_AGENT for the download because it's done in python and doesn't actually use curl. we have a new defaultUSER_AGENTconfig option for exactly this type of situation in the new betas.hydrargyrum commentedon Apr 1, 2025
i'm using
latesttag from https://hub.docker.com/r/archivebox/archivebox/tagshydrargyrum commentedon Apr 1, 2025
yes, but that's what the whole UI says! it shows a curl command! nowhere it's documented that the download is done with python with USER_AGENT!
plus, if there's a download with dom/singlefile/wget, then that one is used, so USER_AGENT is ignored
pirate commentedon Apr 1, 2025
It's not ignored, USER_AGENT is applied to the other methods too. CURL_USER_AGENT just overrides USER_AGENT in the case that any curl command is rendered either for execution or for log display.
The goal is to ensure anytime curl is executed, it has that agent (to avoid ever leaking original agents). The command shown in terminal is a suggested command you can use to debug the behavior, we may use python dotted import function references in the future as well, and in that case we can show the true underlying command, until then a replica curl command that respects CURL_USER_AGENT is good enough.