Geekzone: technology news, blogs, forums
Guest
Welcome Guest.
You haven't logged in yet. If you don't have an account you can register now.

Forums › New Zealand Broadband › Squid Proxy - sites used by time....


1533 posts

Uber Geek

Trusted
Topic # 72123 21-Nov-2010 11:52 Send private message

Hi

running a squid proxy at home, mostly for saving bandwidth in windows updates etc.

But now I'd like to see what is using the internet at certain times to track usage.

Anyone know of a linux install/script/report or something that will allow me to see sites used over specified time period?




Previously knows as psycik

NextPVR Based HTPC:
2 x HVR3000 - DVB-S - Freeview, HVR3000 - DVB-T Freeview|HD, Nova-T 500 - Dual Freeview|HD, Digital Coax --> Yamaha RX-v540, 8600GT --> Samsung LA46A650D via HDMI
Clients:
Popcorn Hour A-100, 1xATV2, 1xATV3
Windows 7 Ultimate Host
4x2TB + 1x1.5TB using DriveBender, VMWare Server 2 with 1xW7, 2xW2k3 1xUbuntu 11.10 Desktop, 1xWHS2011, Plex

Create new topic
164 posts

Master Geek

Trusted
  Reply # 407405 21-Nov-2010 20:07 Send private message

Off http://www.squid-cache.org/Scripts/ is a list of squid log file analysis programs that will tell you what you want to know, .
 



1533 posts

Uber Geek

Trusted
  Reply # 407414 21-Nov-2010 20:52 Send private message

I've got calamaris and webalyser on the system but cannot see it producing the information I want.

Have you used one that will do what I want? if so which one?




Previously knows as psycik

NextPVR Based HTPC:
2 x HVR3000 - DVB-S - Freeview, HVR3000 - DVB-T Freeview|HD, Nova-T 500 - Dual Freeview|HD, Digital Coax --> Yamaha RX-v540, 8600GT --> Samsung LA46A650D via HDMI
Clients:
Popcorn Hour A-100, 1xATV2, 1xATV3
Windows 7 Ultimate Host
4x2TB + 1x1.5TB using DriveBender, VMWare Server 2 with 1xW7, 2xW2k3 1xUbuntu 11.10 Desktop, 1xWHS2011, Plex

21 posts

Geek
  Reply # 408246 23-Nov-2010 14:35 Send private message

I use SARG. It has by far the best summary information of any squid report analysis tool I have tried. It will also draw graphs showing when each machine has visited particular sites, as well as report some of this data numerically.



1533 posts

Uber Geek

Trusted
  Reply # 408302 23-Nov-2010 15:12 Send private message

Yeah I use sarg as well (configured via webmin - be that good or bad).

And while i can get the usage in bytes etc, I can't figure out what i need to set to say xyz machine visited site abc at so and so time.




Previously knows as psycik

NextPVR Based HTPC:
2 x HVR3000 - DVB-S - Freeview, HVR3000 - DVB-T Freeview|HD, Nova-T 500 - Dual Freeview|HD, Digital Coax --> Yamaha RX-v540, 8600GT --> Samsung LA46A650D via HDMI
Clients:
Popcorn Hour A-100, 1xATV2, 1xATV3
Windows 7 Ultimate Host
4x2TB + 1x1.5TB using DriveBender, VMWare Server 2 with 1xW7, 2xW2k3 1xUbuntu 11.10 Desktop, 1xWHS2011, Plex

21 posts

Geek
  Reply # 408465 23-Nov-2010 18:50 Send private message

Are you much of a coder in perl or bash or anything that can convert between unix dates and human-readable dates? When I need specific searches such as yours, I usually do my processing against the original squid log. cat /var/log/squid/access.log | filtercommand > anotheraccess.log

Then, run sarg against the filtered output. In your case, you'd just need a filter than reads the start date/time and end date/time, then outputs only lines that fall between the two.



1533 posts

Uber Geek

Trusted
  Reply # 408801 24-Nov-2010 10:57 Send private message

greaneyr: Are you much of a coder in perl or bash or anything that can convert between unix dates and human-readable dates? When I need specific searches such as yours, I usually do my processing against the original squid log. cat /var/log/squid/access.log | filtercommand > anotheraccess.log

Then, run sarg against the filtered output. In your case, you'd just need a filter than reads the start date/time and end date/time, then outputs only lines that fall between the two.


Not really. I was hoping sarg or webalyser would do it for me.

I've not even really looked at the raw access.log before.




Previously knows as psycik

NextPVR Based HTPC:
2 x HVR3000 - DVB-S - Freeview, HVR3000 - DVB-T Freeview|HD, Nova-T 500 - Dual Freeview|HD, Digital Coax --> Yamaha RX-v540, 8600GT --> Samsung LA46A650D via HDMI
Clients:
Popcorn Hour A-100, 1xATV2, 1xATV3
Windows 7 Ultimate Host
4x2TB + 1x1.5TB using DriveBender, VMWare Server 2 with 1xW7, 2xW2k3 1xUbuntu 11.10 Desktop, 1xWHS2011, Plex

21 posts

Geek
  Reply # 409145 24-Nov-2010 20:39 Send private message

Well do you want it for an ongoing thing, or are you just wanting to know where people are going between certain times just for a little while? If you were after an ongoing thing, then you probably do need something a little less manual, but if you just wanted it for say a week, then it wouldn't be much at all to make something like I was talking about. I could probably throw something together for you if you thought it might help.



1533 posts

Uber Geek

Trusted
  Reply # 409176 24-Nov-2010 21:28 Send private message

if you could, as a manual thing, then I can always hack around with it to make it into a script.

thanks.




Previously knows as psycik

NextPVR Based HTPC:
2 x HVR3000 - DVB-S - Freeview, HVR3000 - DVB-T Freeview|HD, Nova-T 500 - Dual Freeview|HD, Digital Coax --> Yamaha RX-v540, 8600GT --> Samsung LA46A650D via HDMI
Clients:
Popcorn Hour A-100, 1xATV2, 1xATV3
Windows 7 Ultimate Host
4x2TB + 1x1.5TB using DriveBender, VMWare Server 2 with 1xW7, 2xW2k3 1xUbuntu 11.10 Desktop, 1xWHS2011, Plex

Create new topic



Twitter »
Follow us to receive Twitter updates when new discussions are posted in our forums:



Follow us to receive Twitter updates when news items and blogs are posted in our frontpage:



Follow us to receive Twitter updates when new jobs are posted to our jobs board:



Follow us to receive Twitter updates when tech item prices are listed in our price comparison site:




News »
Android and iOS combine for 92.3% of all smartphone OS shipment in the first 2013 quarter
Posted 17-May-2013 08:38

Apple’s App Store marks 50 billionth download
Posted 17-May-2013 08:25

Gen-i divests Davanti Consulting through management buyout
Posted 16-May-2013 14:02

Huawei G510 bring big-screen mobile entertainment for Android users
Posted 16-May-2013 11:49

BlackBerry BBM goes multiplatform: iOS and Android versions
Posted 15-May-2013 11:28

BlackBerry introduces BlackBerry 10 QWERTY smartphone
Posted 15-May-2013 09:32


Trending now »
Hot discussions in our forums right now:

A reason not to shop at dick smith
Created by dsnz1, last reply by AKLWestie on 17-May-2013 22:45 (82 replies)
Pages... 4 5 6

Chorus is cutting the cost of VDSL to service providers from June 7
Created by maxzzz, last reply by Ragnor on 16-May-2013 02:57 (40 replies)
Pages... 2 3

A new project coming to Geekzone
Created by freitasm, last reply by SaltyNZ on 18-May-2013 13:30 (193 replies)
Pages... 11 12 13

HTC One (2013) owners' discussion
Created by Dingbatt, last reply by bradstewart on 18-May-2013 17:45 (1401 replies)
Pages... 92 93 94

Galaxy S4 to run stock Android, by Google
Created by kiwitrc, last reply by Lambchop on 17-May-2013 02:54 (30 replies)
Pages... 2

Sitting on a boring conference call
Created by SaltyNZ, last reply by SepticSceptic on 17-May-2013 16:52 (14 replies)

Samsung Galaxy SIII Discussion and Owners Thread
Created by networkn, last reply by Johnk on 18-May-2013 14:50 (5522 replies)
Pages... 367 368 369

Nokia Lumia 925
Created by motorwayne, last reply by Ragnor on 16-May-2013 03:04 (22 replies)
Pages... 2


Geekzone Jobs »
Most recent NZ jobs in technology:

Office Girl
Posted 18-May-2013 13:27

CRM Lead/ Senior MS CRM Consultant
Posted 18-May-2013 09:27

Business Analyst - Technical Web Focus
Posted 18-May-2013 09:27

Senior Front End Developer
Posted 18-May-2013 09:27

Client Support Analyst
Posted 17-May-2013 23:26

Solution Architect - Global Travel
Posted 17-May-2013 22:26

Analyst Programmer | ATM and Windows Development
Posted 17-May-2013 19:26


Geekzone Live »
Try automatic live updates from Geekzone directly in your browser, without refreshing the page, with Geekzone Live now.

Updates »
Are you subscribed to our RSS feed? You can download the latest headlines and summaries from our stories directly to your computer or smartphone by using a feed reader.

Alternatively, you can receive a daily email with Geekzone updates.