Tools Ways to rip entire TikTok profile?

Obicase

Bathwater Drinker
Mar 16, 2022
61
1,732
1,252
0fya082315al84db03fa9bf467e3.png
Yo, I want to bring attention back to the browser extension Zogbot9001 shared in this this thread two weeks ago.
I had been using this extension since around february, and it's even much more powerful than what Zogbot9001 described.
First, it does get every tiktok from the account you download. The reason you might see a difference between the number at the top and the number of files downloaded is that some toks were deleted from the feed, either by the user or by tiktok ; but if you were to count every tok in the feed (or press Ctrl+Alt+A after scrolling to the bottom), you'd see it really did get everything.

But more importantly, the three most impressive features would be :
1) It can automatically create a new folder for each new account by ticking "Separate folder for downloading profiles" in the extension options, you don't need to set it up manually every time. It also lets you customize the folder name as well as the tiktoks filenames, using a list of pre-set parameters, such as username/user ID/user nickname for folders and video ID/description/date, etc. for video filenames. Its sorting and filenaming capabilities are pretty much perfect. Oh, and also, it gets tiktoks in the highest resolution possible (up to 1080p).
2) It has a "notifier" that downloads every new tiktok from every account you're subscribed to (or a filtered list of accounts that you can set up), completely automatically. That's right. You can let your browser run during the day and this extension will automatically download new tiktoks posted by all of your favorite accounts, just a minute after they're posted (sometimes even getting "spicy" tiktoks that were deleted by tiktok or by the user in the next minutes after posting it). And when you turn off your browser/computer and boot it back up hours or days later, no problem ! The extension will carry on where it stopped and download all of the vids that were posted during the night, weekend, etc.
3) It can download stories and image posts. It can even download them automatically with the notifier.

Personally, I follow hundreds of tiktok accounts. There is no way I could stay up to date with all of them without this browser extension, especially since tiktok's subscription feed doesn't show posts in chronological order (hate when social medias do that).

I could have shared this extension a while ago on this thread, but didn't for a couple reasons. First, the person who initially shared it with me asked me not to share it. Second, the imageboard where the dev of this extension was active has... some weird threads and users. And by weird I mean I feel like I might possibly be breaking the law if I share a link to the site. Probably not since there isn't any content technically illegal per US laws, but it's close.

Anyway, the reasons I'm sharing it now are : 1) while the manual profile scraper of the extension still works, the notifier is broken and the dev doesn't seem to be active anymore (and I really need help fixing it, please). 2) Now that Zogbot9001 already shared it I feel more confortable sharing it too.
So, the thread that this extension was seemingly initially posted in, is this one :
Please, Log in or Register to see links and images

Warning and disclaimer : Stay on the awemer/extension thread. Do not browse the rest of the board. I did not make or do not condone anything on this site.

You'll see at the top of the thread that there is a javascript program called awemer, that was being shared and updated on 4chan and other imageboards since around 2019, that the extension is based on. Awemer has even more features, like being able to scan your folder and compare it to a profile scrape to find exactly which toks are missing. Problem is, the awemer program doesn't have a user interface like the browser extension and I have no fucking clue how to use it. There is a mega link in the thread with the two latest versions of the extension, last one being from may 10th (same thing that Zogbot shared in his post).

Now, what I really wanted to get to with this post, is that the notifier I talked about stopped working on september 19th (11 days ago). I posted the error log in the linked thread. What seems to be the problem is that the API the extension was using, tikwm, was updated that day, and the extension needs to be updated to match that change. That sounds like a really simple problem to fix, so simple that I started learning javescript to try and do it myself, but I quickly realized it would actually take me months to be able to get there.

To conclude, if anyone in this thread knows javascript, this extension/program is defintely the way to go. Forget any other program you've been using so far to download from tiktok because this one is the best, but it just needs a little fix. If anyone could find a fix for the notifier, it would be awesome. Alternatively, if anyone could post a tutorial on how to use the awemer javascript program to mimic what the extension did, it would be equally awesome.
I'll be trying to learn and find a fix on my own in the meantime but I can't guarantee anything. I never learned how to code so I'm learning from scratch for this.
 
Last edited:
  • Like
Reactions: jasonjemodah
Mar 19, 2022
3
3
58
0fya082315al84db03fa9bf467e3.png
Please, Log in or Register to view quotes
Do you know if the extension is open source? For example, is there a GitHub repository for it, or is the source for the extension accessible anywhere? If so, then I can probably fix whatever issues there are with the extension. If not, it is significantly (multiple orders of magnitude) more challenging to even consider how to approach fixing compiled JS code.
 

Obicase

Bathwater Drinker
Mar 16, 2022
61
1,732
1,252
0fya082315al84db03fa9bf467e3.png
Please, Log in or Register to view quotes
The dev of the extension did have a github page for it, but there was only a readme there, no source code. It was the same readme you get when downloading the extension on the mega link. The github page is deleted now though.

When you say compiled JS code, are you referring to the aes.js file that seems to be ecrypted with cryptoJS ? Admittedly I have no idea how to decrypt that, but I wouldn't be surprised if there are tools that can decrypt it, the same way there are tools that can deobfuscate code that was obfuscated. The other .js files seems to be human readable. If that's actually hard to decrypt, then you are right, that sucks.

If that's any help, the awemer program is open source. You can download the code in the first post of the thread I linked and install it with node. No clue how to use it though, I tried the beta python UI but didn't get it to work.
 

Obicase

Bathwater Drinker
Mar 16, 2022
61
1,732
1,252
0fya082315al84db03fa9bf467e3.png
Hi, I am sharing another python script made by an Anon that kindly authorized me to share it here. I wasn't sure if he wanted his mega link posted somewhere else so I reuploded it here :
Please, Log in or Register to see links and images

This is a command line program. Pretty straight foward to use if you have already used similar scripts before (like ytp-dl that was posted in this thread). Still, if someone needs it I can make a step by step guide.
It is open source so feel free to check the code. I've been using it for a couple weeks now, no problem.

It also needs ffprobe, python, and the "requests" library. Pretty easy, more info in the README.txt or Requirements.txt.

This script cannot download stories, image posts, or private accounts (you can't give it your cookies).

However, besides those three missing features, it is still really powerful. The command at its simplest looks like this :
python awemer.py -u not.papadonks
This will download all of @not.papadonks 's videos in the highest resolution possible, in a folder named not.papadonks in the directory where the program is (if the folder doesn't exist, it will create it). The default filename for videos is VIDEO_ID.mp4, or VIDEO_ID~sd.mp4 if the video is low res (576x1024 or 1024x576).

Alternatively, you can feed it a list of usernames in a text file.
python awemer.py -u profiles.txt
Will download all of the videos of all of the profiles in profiles.txt (one username per line), each in their respective folder.

You can also use user ID instead of usernames, ie : python awemer.py -u 6998772783719285766

There are several parameters that you can add.
For exemple, -t to download all of the videos released after a certain date. For exemple, this will download every video released after the 15th of october 2023.
python awemer.py -u not.papadonks -t 2023-10-15

-u to change the directory, -p to use http proxy, -s to add video description to the filenames, -i to print user ID and secID, -ad to add the date to the filenames, -se to skip existing SD videos, -l to loop.
For more details on each parameter, check the readme in the mega link.

You can also use the executable instead of python. The command line then looks like this :
awemer.exe -u not.papadonks

I personally use this :
python awemer.py -u profiles.txt -t [yesterday's date]
"profiles.txt" contains all of the profiles that I follow (around 700), and I set the date to yesterday's date. I run this two or three times a day. This way, this downloads automatically every video released by all 700 accounts in the list, without me having to do anything.

Note that the API limit is 5000 requests per day. This is the case for all tiktok scrapers, not just this one. This is why there is an option for proxies. Like I said, I update 700 accounts per day, three times a day, and I don't hit the limit. The only time you might hit it is if you do a scrape of a large number of accounts with no -t parameter.
 

JuicyLucy666

Bathwater Drinker
Mar 3, 2022
92
3,135
1,242
0fya082315al84db03fa9bf467e3.png
Please, Log in or Register to view quotes
Well, I just tried it on an account with more videos and now I keep getting this error "Error: ('Connection broken: IncompleteRead(3229912 bytes read, 1633014 more expected)', IncompleteRead(3229912 bytes read, 1633014 more expected))"
Have you encountered such an error, or know how I can possibly bypass it?
 

Obicase

Bathwater Drinker
Mar 16, 2022
61
1,732
1,252
0fya082315al84db03fa9bf467e3.png
Please, Log in or Register to view quotes
First thing you can try is to delete the program and redownload it (just the program, not the accounts you already downloaded). That fixed a similar problem I had gotten after stopping a scrape abruptly with Ctrl+C.

If that doesn't help, I'll need more info.
Is it only this account that is giving this error, have you tried other accounts after getting the error ?
Does the error appear immediately after running the command of after some time downloading ?
Could you maybe post the account username and the full error log ?
 

Beakwood

Bathwater Drinker
Mar 11, 2022
631
5,474
1,217
0fya082315al84db03fa9bf467e3.png
The one drawback, for me so far, is the "Has No Content". When I clearly see multiple videos on the user's page. Could be a possibility of the account being for more adults. But, even "PG" rated users, are suffering from the same. For example,
Please, Log in or Register to see links and images
I would say is now more PG and have some videos on profile, but the tool says "HAS NO CONTENT". Still pretty good.
 

Obicase

Bathwater Drinker
Mar 16, 2022
61
1,732
1,252
0fya082315al84db03fa9bf467e3.png
Please, Log in or Register to view quotes
That's an instagram. Did you mean to link something else ?

Edit : to clarify, the tool I linked can only download tiktok profiles. If you tried to download an account called " lex_p_ ", then it tried to download the tiktok profile with this username, which has no content.
 
Last edited:

Beakwood

Bathwater Drinker
Mar 11, 2022
631
5,474
1,217
0fya082315al84db03fa9bf467e3.png
Please, Log in or Register to view quotes
Complete mistake. LOL! But it does work for Instagram profiles. I've been following Instagram and TikTok threads and thought this was Instagram. But, Yes, This does indeed work for Instagram as well (somewhat), which is interesting.
As for TikTok, I've been using yt-dlp still. Granted, it works with plug-in. Just add the "
Please, Log in or Register to see links and images
" plug-in and should be good to go, for those that's been using yt-dlp method.
 
Last edited:

Holla33

Superfan
Mar 11, 2022
35
576
739
0fya082315al84db03fa9bf467e3.png
Please, Log in or Register to view quotes
Obicase First thing thanks, what you did works very well thank you! You did a good work

So i have the same problem as the other dude which is:
Please, Log in or Register to view quotes
So the last video it downloaded was "2023-05-15 7233217149634891050" but then i had a problem with my connection (nothing to do with your program) and the program stopped so i relaunched the program and now it managed to download the rest. Problem is that i tried to download 40 account from tiktok and i couldn't because of my connection problem lol but the error i got was not only on that account.

So to answer the question you've asked to that other dude if that can help
Please, Log in or Register to view quotes
- It's not the only account giving that error i had it on some other accounts but it was random tbh
- After some time downloading. Sometimes it's like 20 vids before it stops and then sometimes its less than that (for one account it was 1 download and then error
-I'm redownloading rn but i will give you the account names later, for the error log idk where it is lol
 

Obicase

Bathwater Drinker
Mar 16, 2022
61
1,732
1,252
0fya082315al84db03fa9bf467e3.png
JuicyLucy666 Holla33

The anon that develops the program uploaded a new version in which the bug seems to have been fixed. I updated the mega link.

Try the .exe in the v9 folder or the most recent python in the main folder and let me know if you still have an issue.
 
  • Like
Reactions: jasonjemodah

Obicase

Bathwater Drinker
Mar 16, 2022
61
1,732
1,252
0fya082315al84db03fa9bf467e3.png
Updated the mega link.
- Can download image posts
- Can download multiple videos in parallel
- Can notify of new toks of accounts in your list
- Bug fixes

Refer to readme to see how to use these new parameters.

Thanks to anon who develops this program. This is quickly surpassing every other tiktok downloader I've seen.
 

Holla33

Superfan
Mar 11, 2022
35
576
739
0fya082315al84db03fa9bf467e3.png
Please, Log in or Register to view quotes
Sorry again don't want to bother you but since idk the person who made this program you are the only person for the bug report lol , but the latest update made the program unusable. i thought it was on my end but i can confirm the program works (for me at least) in v9, 10 and 11 but not for the recent one.
 

Holla33

Superfan
Mar 11, 2022
35
576
739
0fya082315al84db03fa9bf467e3.png
Please, Log in or Register to view quotes
I use the python, i use these arguments and command line python awemer.py -u profiles.txt -ad -md 4 -dp and when it tries to download the video it says: ... is corrupted. Deleted. | Exception: [WinError 2] The specified file cannot be found

And when i tried the old version it works (without -md and -dp since it wasn't there) and i tried it without the new arguments and i still have a problem.
 

Obicase

Bathwater Drinker
Mar 16, 2022
61
1,732
1,252
0fya082315al84db03fa9bf467e3.png
Please, Log in or Register to view quotes
You need ffprobe in your PATH, it's mentionned in the readme.

Download it here :
Please, Log in or Register to see links and images

If you're on windows x64 it's ffmpeg-master-latest-win64-gpl.zip
I don't think it matters where you put it but I guess you can create a folder and extract it in C:\Users\USER\AppData\Local\Programs\ffmpeg

Then you add C:\Users\USER\AppData\Local\Programs\ffmpeg\bin to PATH.
Tutorial :
Please, Log in or Register to see links and images


Then restart your computer and try again.
 

xoeh

Bathwater Drinker
Sep 21, 2023
113
5,940
532
0fya082315al84db03fa9bf467e3.png
Has anyone figured out how to download videos if a profile is private but you're still following said account?
I realized both awemer and the chrome-awemer versions don't work when someone privates their account, despite you being an authorized account/follower.

Thanks Obicase for the awemer python version; sadly I could not find any options to incorporate an exported browser cookie.

(resolved issue about WinError2 above w/ ffprobe, derp; but still curious if anyone knows how to download with followed private accounts)
 
Last edited: