Debian Planet











Welcome to Debian Planet

Search

Apt-get into it.
Main Menu

  • Home

  • Topics

  • Web Links

  • Your Account

  • Submit News

  • Stats

  • Top 10

  • Debian

    These are important Debian sites one should not be without!

  • Official Debian site

  • Package search

  • Mailing list archives

  • Bug reports

  • Debian on CD

  • Unofficial woody CD ISOs

  • Unofficial APT sources

  • Developers' Corner

    Other great Debian news sources:

  • Debian Weekly News

  • Kernel Cousin Debian

    (Debian mailing lists digested)
  • Community Groups

    Need help? You're not alone on this planet.

  • debianHELP

    (User support site)

  • Debian International

  • DebianWorld

    (Français)

  • DebianForum.de

    (Deutsch)

  • EsDebian

    (Español)

  • Debian-BR

    (Português)

  • DebianUsers

    (Korean)

  • IRC

    The place to get help on a Debian problem (after reading docs) or to just chat and chill is #debian on irc.debian.org.

    Many of the Debian Planet staff live there so pop by and say hello.

    Wanna write?

    Got that latest or greatest scoop? Perhaps you have some important news for the Debian community? Submit a news item!

    Or perhaps you've written a rather ground breaking insight into some aspect of Debian and you feel compelled to share it with others? Knock up a longer editorial article and send it to our team.

    Sponsorship

    DP is sponsored by uklinux.net and CheepLinux.

    Debian Planet runs on hardware donated by Xinit systems and is using kieser.net's bandwidth.

    Who's Online

    There are currently, 66 guest(s) and 6 member(s) that are online.

    You are Anonymous user. You can register for free by clicking here.

      
    Caching apt mirror?
    Contributed by crazney on Saturday, September 08 @ 09:08:02 BST

    Ask Debianplanet
    I have between 3 and 5 debian boxes at home at any one time, all running SID. I often do dist-upgrades on all of them, which as you can imagine, shoves my bandwdith costs up a bit.. Does anyone know of a nice way to 'cache' my apt-get's so I only need to get each package once for my entire network (a full on http/ftp proxy is out of the question).

    DanielS: apt-proxy/apt-move. BTW, why is Squid out of the question?

     
    Related Links

  • More about Ask Debianplanet
  • News by DanielS

    Most read story about Ask Debianplanet:
    Why isn't Debian 'ported' to the higher x86 machines?

    Last news about Ask Debianplanet:

    Printer Friendly Page  Send this Story to a Friend
  • "Caching apt mirror?" | Login/Create Account | 14 comments
    Threshold


    The comments are owned by the poster. We aren't responsible for their content.

    Re: Caching apt mirror? (Score: 0)
    by Anonymous on Saturday, September 08 @ 09:31:28 BST

    I'm sorry I can't really comment on the orginal question, but I have a related question of my own:

    How much space do you need for a (full) Debian mirror and/or what different options do I have in setting up such a mirror? Is there a simple howto/faq anywhere?

    I'm considering setting up such a mirror for my fellow (Debian-using) students. Every student in the building has a 100Mbit connection, but you don't often get the maximum from the mirrors and it would generally better for the connection to Internet if things like this would be locally available.

    [ Reply ]


    Re: Caching apt mirror? (Score: 0)
    by Anonymous on Saturday, September 08 @ 12:54:52 BST

    Possible solutions:

    - Set up a FTP server and point your clients to your server's address+path, don't forget to run dpkg-scanpackages everytime you 'apt-get dist-upgrade' on the server

    - NFS and make sure /var/cache/apt/archives is one of your exports

    - Run a proxy (why is it out of question?)

    [ Reply ]


    Re: Caching apt mirror? (Score: 1)
    by Anonymous on Saturday, September 08 @ 13:13:59 BST

    Although apt-proxy is pretty good, it does have some issues, you never know what status your download is in and when it gets to 99% you need to restart apt-get to finish off. It also generates several dead (or avtive) sleep and rsync processes. Anything more that 3 or 4 machines apt-proxy is probably the wrong solution.

    [ Reply ]


    Re: Caching apt mirror? (Score: 1)
    by DanielS on Saturday, September 08 @ 13:45:37 BST
    (User Info) http://kabuki.sfarc.net/daniel/

    Hmm, then your best option probably *is* a source. Here's a quick primer:

    cd /var/www

    ln -s /var/cache/apt/archives /var/www/debian

    cd debian

    dpkg-scanpackages . /dev/null > Packages && gzip -9c Packages > Packages.gz

    Then:

    deb http://foo.bar.baz/debian .

    🙂 d

    [ Reply ]


    Re: Caching apt mirror - Just use RSync? (Score: 2, Interesting)
    by benmhall on Saturday, September 08 @ 15:41:43 BST
    (User Info) http://www.moses.cx

    Hi there,

    At the school lab the systems have been configured to update files from a central machine. One system is maintained and runs an rsync server, all other machines query this machine and pull down differences. We had to set up which files are unique (IP settings, hostname and a slew of others) but this has worked very well in a heavily abusive public area for several years now.

    Best of all, it was pretty easy to set up! Just configure the rsync server machine (read the man page) and set up a cron job on each machine to pull the differing files from the server.

    Of course, this system works for far more than just Debian, but doesn't take advantage of apt in any way... In fact, the machines in question are running Mandrake 6.0. You could use the same scheme for just about anything, though. (I looked into setting something similar up for Solaris as well..) rsync runs on just about everything..

    Ben

    [ Reply ]


    Re: Caching apt mirror? (Score: 2, Informative)
    by camh on Sunday, September 09 @ 01:35:50 BST
    (User Info)

    I've got only two sid systems and couldn't manage without any caching. I
    dont know how you do five. My two systems are my laptop which I take
    to work, and my desktop at home which is stuck behind a 56k modem. I
    do most of my apt-getting at work because its got a faster link. Here's
    how I do it:

    On my laptop I have a /mirror/debian. I use apt-move to move the packages
    from the apt cache into the mirror hierarchy. My sources.list has the
    mirror as the first entry on each machine. I NFS mount the mirror from
    my laptop to my desktop. When I apt-get on my desktop, it will first
    look in the mirror and only if the package is not there will it go out
    via the modem.

    A typical session on my laptop looks like this:


    $ apt-get update
    ...
    $ apt-get -du dist-upgrade
    ...
    # apt-get dist-upgrade
    ...
    $ apt-move update
    ...
    $ apt-get update

    I do most of this as myself, not root. Only the actual installation of
    pacakages are done as root. The last apt-get update is to make sure the
    lists for the local mirror are up-to-date.

    On my desktop, it looks like this:


    $ cd /var/lib/apt/lists
    $ scp -P laptop:/var/lib/apt/lists/*_* .
    $ mount /mirror/debian
    $ pon
    $ apt-get -du dist-upgrade
    ...
    $ poff
    # apt-get dist-upgrade
    ...
    $ apt-move update
    $ umount /mirror/debian

    I dont do an apt-get update on my desktop, because I'm too impatient
    for 3mb of lists to download via 56k modem. The sources.list on both
    machines are synchronised, so I can just grab the apt lists from my
    laptop without needing to update. Most packages on both machines are
    shared, so the vast majority are pulled down from the net over my work
    link to my laptop. The only extra stuff I have on my desktop are a few
    tv applications (xawtv...), so my typical upgrade size for my desktop
    is 1-2MB.

    The key is apt-move. I think that's what you're looking for.

    [ Reply ]


    Re: Caching apt mirror? (Score: 1)
    by hn75 on Sunday, September 09 @ 17:36:11 BST
    (User Info)

    I have written a script that mirrors an certain

    dist. Send me an email (hn75@gmx.de) and I will return it

    to you.

    Hendrik

    [ Reply ]


    Re: Caching apt mirror? (Score: 0)
    by Anonymous on Monday, September 10 @ 11:47:54 BST

    I just use the proxy-cache from my ISP (in

    this case it is my University and the academic

    network cascaded global cache).


    If you set (in /etc/environment) the var

    http_proxy=yourproxyhere (and also ftp_proxy)

    then apt-get (and

    dselect) will use the proxy. With a decently

    configured proxy you will only donwload once

    the new files (and just only those files required)

    and (as in my case) shared with other Debian users

    attached to the same proxy.


    If your ISP does not maintain a proxy-cache, then

    you can install one for yourself.

    [ Reply ]


    Re: Caching apt mirror? (Score: 1)
    by Anonymous on Monday, September 10 @ 22:15:17 BST

    do this on the mirror/server (in a script of course):

    o apt-get update

    o apt-get upgrade -qq --no-holds --print-uris (or something like that, i don't currently have access to the script)

    o sed out deb uris based on above apt-get output

    o wget -mv debs into local mirror based on above sed output

    o dpkg-scanpackage local mirror

    o gzip above dpkg-scanpackage output

    o occasionally compare mirror vs cached Packages list in /var/cache/apt/... and delete outdated packages (packages that are in the mirror but not in any Packages list)

    to make sure your cache gets depended on, either make sure your local cache appears above all other sources (in sources.list) or play with your preferences file.

    i created the above script not to share the local mirror with other machines, but so that i could download these files (at 4AM) and upgrade from them at my convenience, have the packages avaiable incase i have to reinstall one of them, and have a perfect (though incomplete) mirror. now that i have a laptop, i use the local repository to update it.

    the short-coming of this script: the far mirrors are updated 4 times a day but the local mirror is only updated once, so if the laptop does an "apt-get update" in the evening (after the far mirrors have updated but the local mirror hasn't), then the laptop will grab any updated packages from the far server. I haven't thought out how to get those packages on the laptop to the local mirror (so that the local mirror won't download them as a local computer already has the package).

    and yeah, i mentioned this script in a previous posting a month or so back and said that i would put the script up somewhere for downloading. well, i never did (as i don't really have anywhere public to put it). but no one emailed me about it (though, dang it, debianplanet keeps posting my comments as from "anonymous" though i'm logged in and i get my default +1 mod), so i figure no one wanted it that badly. 😉

    [ Reply ]


    Based on: PHP-Nuke

    All logos and trademarks in this site are property of their respective owner. The comments are property of their posters, all the rest © 2000 by Debian Planet

    You can syndicate our news using the file backend.php.