Difference between revisions of "SitePerformanceMonitoringTools"

(→‎IP connectivity [R])
 
Rating: 0 - 0 votes

Company Name

Company Contact

Page Type

This page is about a company.

(11 intermediate revisions by 5 users not shown)



Line 1: Line 1:
−
<noinclude><big>[[OurWork]] < [[DevelopmentTeam]] < [[DevelopmentTeamPriorities|Priorities]] < </noinclude>('''10''') [[SitePerformanceMonitoringTools]] ('''[[Ethan]]''') {{JustTinyEditIcon|SitePerformanceMonitoringTools}}<noinclude></big>
+
<noinclude><big>[[OurWork]] < [[DevelopmentTeam]] < [[DevelopmentTeamPriorities|Priorities]] < </noinclude><strike>('''[[PairDays|4]]''') [[SitePerformanceMonitoringTools]] ('''[[Ethan]]''') {{JustTinyEditIcon|SitePerformanceMonitoringTools}}</strike><noinclude></big>
  
 
== What (summary) ==
 
== What (summary) ==
 
Instrumentation that provides a history of performance statistics for each part of the page load pipeline.
 
Instrumentation that provides a history of performance statistics for each part of the page load pipeline.
 +
 +
Related pages:
 +
* [[AboutUsPerformanceMonitoring]]
 +
* [[SystemMonitoringDashboard]]
  
 
== Why this is important ==
 
== Why this is important ==
Line 8: Line 12:
  
 
== [[DoneDone]] ==
 
== [[DoneDone]] ==
−
* A Dashboard with Red, Yellow or Green for each pipeline item.
+
* :A Dashboard with Red, Yellow or Green for each pipeline item.
−
* The history for each pipeline item stored in the database.
+
* :The history for each pipeline item stored in the database.
−
* Definitions of acceptable benchmarks for each pipeline item.
+
* :Definitions of acceptable benchmarks for each pipeline item.
−
* Definitions of acceptable freshness for each pipeline item.
+
* :Definitions of acceptable freshness for each pipeline item.
−
* All monitoring meets the definitions for minimum freshness.
+
* :All monitoring meets the definitions for minimum freshness.
  
 
==Performance Priorities==
 
==Performance Priorities==
Line 29: Line 33:
  
 
==Steps to DoneDone==
 
==Steps to DoneDone==
−
* Articulate the request pipeline
+
* <strike>Articulate the request pipeline</strike>
 
** <strike>Identify each request in the pipeline</strike>
 
** <strike>Identify each request in the pipeline</strike>
−
** '''Friday''' ... How to perform / retrieve data for each request
+
** <strike>How to perform / retrieve data for each request</strike>
−
* Aggregate pipeline benchmarks
+
* <strike>Aggregate pipeline benchmarks</strike>
−
** '''Tuesday''' ... Implement the probes
+
** <strike>'''Tuesday''' ... Implement the probes</strike>
−
** '''Wednesday portland devs''' ... Articulate database schema
+
** <strike>'''Wednesday portland devs''' ... Articulate database schema</strike>
−
** '''Wednesday portland devs''' ... Push XML results to central HTTPS server
+
** <strike>'''Wednesday portland devs''' ... Push XML results to central HTTPS server</strike>
−
** '''Wednesday portland devs''' ... Remote location / benchmark results stored in database
+
** <strike>'''Wednesday portland devs''' ... Remote location / benchmark results stored in database</strike>
−
* Integrate into monitoring
+
** Project revised for local monitoring only using Zabbix. Service on nimbus communicates with agents installed on each server and stored in MySQL.
−
** Dashboard to identify overall health
+
* <strike>Integrate into monitoring</strike>
−
** Notifications via email/paging critical problem arises
+
** <strike>Dashboard to identify overall health</strike>
−
* Analyze pipeline benchmarks
+
** <strike>Notifications via email/paging critical problem arises</strike>
−
** XML output for monitoring integration
+
** Available at https://admin.aboutus.org/zabbix/
−
** Graph performance for each location
+
* <strike>Analyze pipeline benchmarks</strike>
−
** Detailed graph view on each request
+
** <strike>XML output for monitoring integration</strike>
−
* Define acceptable benchmarks for each request
+
** <strike>Graph performance for each location</strike>
 +
** <strike>Detailed graph view on each request</strike>
 +
** Information available at http://www.aboutus.org/AboutUsPerformanceMonitoring/
 +
* <strike>Define acceptable benchmarks for each request</strike>
 +
 
 +
== Repositories ==
 +
* cd into the directory you want to check the client out into
 +
* git clone nimbus:/opt/git/geophone-client
 +
* make a few changes in that directory
 +
* git status ... to see what is different
 +
* git diff ... to go through the change and make sure you aren't including something accidentally
 +
* git add YYYY ... to include modified or new file YYYY in the commit
 +
* git commit -m 'Here is why I made the changes I did for this commit'
 +
* git push ... to make sure that the remote repository has your changes
 +
* gitk ... from within the directory shows the tree of revisions
  
 
==Pipeline==
 
==Pipeline==
Line 65: Line 83:
 
: dig images.aboutus.org @ns2.dnscloud.com
 
: dig images.aboutus.org @ns2.dnscloud.com
  
−
==== DNS Results ====
+
'''Results'''
−
 
 
 
* www.aboutus.org
 
* www.aboutus.org
 
: ns1.dnscloud.com
 
: ns1.dnscloud.com
 
:: IP Address
 
:: IP Address
−
:: Query time
+
:: Query time (ms)
 
: ns2.dnscloud.com
 
: ns2.dnscloud.com
 
:: IP Address
 
:: IP Address
−
:: Query time
+
:: Query time (ms)
  
 
* images.aboutus.org
 
* images.aboutus.org
 
: ns1.dnscloud.com
 
: ns1.dnscloud.com
 
:: IP Address
 
:: IP Address
−
:: Query time
+
:: Query time (ms)
 
: ns2.dnscloud.com
 
: ns2.dnscloud.com
 
:: IP Address
 
:: IP Address
−
:: Query time
+
:: Query time (ms)
−
 
 
−
=== IP Connectivity [R] ===
 
  
 +
=== IP Connectivity ===
 
Network connectivity and latency can be measured using ping and traceroute utilities. Most issues with connectivity will most likely be caused by network problems between the two locations which we have no control over. In some cases, the issues could be caused by router, switch, or load-balancer issues on the AboutUs side, but these items will affect all remote locations.
 
Network connectivity and latency can be measured using ping and traceroute utilities. Most issues with connectivity will most likely be caused by network problems between the two locations which we have no control over. In some cases, the issues could be caused by router, switch, or load-balancer issues on the AboutUs side, but these items will affect all remote locations.
  
Line 90: Line 106:
 
: traceroute -n www.aboutus.org
 
: traceroute -n www.aboutus.org
  
−
==== IP Results ====
+
'''Results''' - <i>Five ICMP packets</i>
−
<i>Five ICMP packets</i>
 
  
−
* packet loss (%) - 5 packet request
+
* packet loss (%)
 
* average response time (ms)
 
* average response time (ms)
  
Line 99: Line 114:
 
<i>/index.php and requisite pages</i>
 
<i>/index.php and requisite pages</i>
  
−
: Response time of the frontpage request is critical. Performance relies on a number of factors.
+
Response time of the frontpage request is critical. Performance relies on a number of factors.
  
 
* Physical server load
 
* Physical server load
Line 138: Line 153:
 
** 3 < t sec is unacceptable
 
** 3 < t sec is unacceptable
  
−
==== HTTP Frontpage Results ====
+
'''Results'''
 
* Request time (ms)
 
* Request time (ms)
  
 
=== HTTP Search ===
 
=== HTTP Search ===
−
 
 
Response time for search results to be displayed displayed.
 
Response time for search results to be displayed displayed.
 
: curl --location --form auSearch=aboutus.org http://www.aboutus.org/Special:AboutUsSearch
 
: curl --location --form auSearch=aboutus.org http://www.aboutus.org/Special:AboutUsSearch
  
−
==== HTTP Search Results ====
+
'''Results'''
 +
* Request time (ms)
 +
 
 +
=== HTTP Random ===
 +
Response time for a random page to be displayed.
 +
: curl --location --silent  http://www.aboutus.org/Special:Random
 +
 
 +
'''Results'''
 
* Request time (ms)
 
* Request time (ms)
  
−
=== HTTP random page ===
+
=== HTTP/HTTPS Authentication ===  
 +
Measure the time it takes for a user to log into the www.aboutus.org site.
  
−
: Response time for a random page to be displayed.
+
: curl --location --cookie-jar - --form wpName=user\ name --form wpPassword=password http://www.aboutus.org/index.php?title=Special:Userlogin\&action=submitlogin\&type=login\&returnto=Wiki
−
: curl --location --silent  http://www.aboutus.org/Special:Random
+
: curl --location --insecure --cookie-jar - --form wpName=user\ name --form wpPassword=password https://www.aboutus.org/index.php?title=Special:Userlogin\&action=submitlogin\&type=login\&returnto=Wiki
  
−
=== HTTP/HTTPS user login ===
+
'''Results'''
 +
* Request time (ms)
  
−
: Measure the time it takes for a user to log into the www.aboutus.org site.
+
=== HTTP Edit ===
 +
: http://www.aboutus.org/index.php?title=domain.com&action=edit
  
−
:  curl --location --cookie-jar - --form wpName=user\ name --form wpPassword=password http://www.aboutus.org/index.php?title=Special:Userlogin\&action=submitlogin\&type=login\&returnto=Wiki
+
'''Results'''
−
:  curl --location --insecure --cookie-jar - --form wpName=user\ name --form wpPassword=password https://www.aboutus.org/index.php?title=Special:Userlogin\&action=submitlogin\&type=login\&returnto=Wiki
+
* Request time (ms)
  
−
=== HTTP page edit ===
+
=== HTTP Save ===
−
http://www.aboutus.org/index.php?title=domain.com&action=edit
+
: edittime=`curl --location http://www.aboutus.org/index.php?title=ObsidiansAnd.com\&action=edit | grep wpEdittime | awk -F\" '{print $2}'`<p>
 +
: curl --location --form wpSave=Save\ page --form wpTextbox1=replacement\ text --form wpEditToken=\\ --form wpEdittime=$edittime http://www.aboutus.org/index.php?title=ObsidiansAnd.com\&action=submit
 +
: <i>warning: will replace all article body with text in wpTextbox1</i>
  
−
=== HTTP edit page save ===
+
'''Results'''
−
warning, the below will replace all article body with text in wpTextbox1.
+
* Request time (ms)
  
−
edittime=`curl -s -L http://www.aboutus.org/index.php?title=ObsidiansAnd.com\&action=edit | grep wpEdittime | awk -F\" '{print $2}'`<p>
+
=== HTTP Render Invalidated ===
−
curl -L --form wpSave=Save\ page --form wpTextbox1=replacement\ text --form wpEditToken=\\ --form wpEdittime=$edittime http://www.aboutus.org/index.php?title=ObsidiansAnd.com\&action=submit
+
: curl http://www.aboutus.org/Wiki -d action=purge
  
−
=== Render invalidated frontpage ===
+
'''Results'''
 +
* Request time (ms)
  
 
==Potential Hurdles==
 
==Potential Hurdles==
Line 189: Line 216:
  
  
 +
</noinclude>
 
[[Category:DevelopmentTeamProject]]
 
[[Category:DevelopmentTeamProject]]
−
</noinclude>
 

Latest revision as of 05:44, 19 October 2007

Rating: 0 - 0 votes

Company Name

Company Contact

Page Type

This page is about a company.

OurWork (4) SitePerformanceMonitoringTools (Ethan)
Edit-chalk-10bo12.png

What (summary)

Instrumentation that provides a history of performance statistics for each part of the page load pipeline.

Related pages:

Why this is important

The responsiveness and performance of the site makes a big difference in how many pages visitors will view, and how often they will come back. A poorly performing site will also wear out our active members causing some of them to leave.

DoneDone

  •  :A Dashboard with Red, Yellow or Green for each pipeline item.
  •  :The history for each pipeline item stored in the database.
  •  :Definitions of acceptable benchmarks for each pipeline item.
  •  :Definitions of acceptable freshness for each pipeline item.
  •  :All monitoring meets the definitions for minimum freshness.

Performance Priorities

  1. View normal page
  2. View random page
  3. Edit click until available
  4. Save click until rendered
  5. Render invalidated frontpage

Instrumentation Steps

=== # End to end on each

  1. Deploy instrumentation boxes in various locations
  2. Determine and instrument the pieces
    • MediaWiki profiling
    • Raw database queries

Steps to DoneDone

  • Articulate the request pipeline
    • Identify each request in the pipeline
    • How to perform / retrieve data for each request
  • Aggregate pipeline benchmarks
    • Tuesday ... Implement the probes
    • Wednesday portland devs ... Articulate database schema
    • Wednesday portland devs ... Push XML results to central HTTPS server
    • Wednesday portland devs ... Remote location / benchmark results stored in database
    • Project revised for local monitoring only using Zabbix. Service on nimbus communicates with agents installed on each server and stored in MySQL.
  • Integrate into monitoring
  • Analyze pipeline benchmarks
  • Define acceptable benchmarks for each request

Repositories

  • cd into the directory you want to check the client out into
  • git clone nimbus:/opt/git/geophone-client
  • make a few changes in that directory
  • git status ... to see what is different
  • git diff ... to go through the change and make sure you aren't including something accidentally
  • git add YYYY ... to include modified or new file YYYY in the commit
  • git commit -m 'Here is why I made the changes I did for this commit'
  • git push ... to make sure that the remote repository has your changes
  • gitk ... from within the directory shows the tree of revisions

Pipeline

DNS request - www.aboutus.org & images.aboutus.org

  • Local resolver / cache

Queries against the resolvers at the remote location provides little insight into health of the www.aboutus.org site. If the record does not exist in the local resolver cache (or the TTL has expired), the DNS root servers will be contacted and the authoritative servers. If the record already exists in the cache then it will respond immediately. If the local resolver does not reply as expected, then the issue likely lies with the remote location or possibly the authoritative name servers (or somewhere between).

  • Authoritative name server
 ns1.dnscloud.com
 ns2.dnscloud.com

Response time of the authoritative server is critical. This can also be measured from any location, though, network latency and connectivity will be a factor.

dig www.aboutus.org @ns1.dnscloud.com
dig www.aboutus.org @ns2.dnscloud.com
dig images.aboutus.org @ns1.dnscloud.com
dig images.aboutus.org @ns2.dnscloud.com

Results

  • www.aboutus.org
ns1.dnscloud.com
IP Address
Query time (ms)
ns2.dnscloud.com
IP Address
Query time (ms)
  • images.aboutus.org
ns1.dnscloud.com
IP Address
Query time (ms)
ns2.dnscloud.com
IP Address
Query time (ms)

IP Connectivity

Network connectivity and latency can be measured using ping and traceroute utilities. Most issues with connectivity will most likely be caused by network problems between the two locations which we have no control over. In some cases, the issues could be caused by router, switch, or load-balancer issues on the AboutUs side, but these items will affect all remote locations.

ping -c 5 -i 0.2 -q www.aboutus.org
traceroute -n www.aboutus.org

Results - Five ICMP packets

  • packet loss (%)
  • average response time (ms)

HTTP Frontpage

/index.php and requisite pages

Response time of the frontpage request is critical. Performance relies on a number of factors.

  • Physical server load
    • CPU
    • Disk I/O
    • Available memory - too little memory causes swapping thereby causing disk I/O performance degradation
    • Network throughput
  • Apache process performance
    • CPU usage
    • Available threads
  • Memcached
    • Down cache / timeout
  • Database query
    •  ?Query for each /index.php request?
    • physical DB (slave) server loads
    • replication
    • MySQL performance
  • DNS request - images.aboutus.org
  • Image GET request
    • image size
    • number of images per page
    • NFS server load (disk I/O)
    • network throughput
curl --silent --write-out %{time_total} --output > /dev/null http://www.aboutus.org/index.php | tail -1
write-output variables: http_code time_total time_namelookup size_download speed_download

Acceptable Benchmarks

Max cold-request to fully rendered time for front page

    • 0
curl --location --form wpSave=Save\ page --form wpTextbox1=replacement\ text --form wpEditToken=\\ --form wpEdittime=$edittime http://www.aboutus.org/index.php?title=ObsidiansAnd.com\&action=submit
warning: will replace all article body with text in wpTextbox1

Results

  • Request time (ms)

HTTP Render Invalidated

curl http://www.aboutus.org/Wiki -d action=purge

Results

  • Request time (ms)

Potential Hurdles

  • False positives
  • Caching

Questions

Does the pipeline include all of these? Record a history of how long to

  • lookup DNS for www.aboutus.org, images.aboutus.org, ... from different parts of the world
  • Setup a port 80 TCP connection with each of the squal boxes from different parts of the world
  • Load the frontpage without any client caching
  • Retrieve a memcache item from each combination of two squal boxes (one client, one memcached server)
  • Load the core css files
  • Load the core js files

...



Rating: 0 - 0 votes

Company Name

Company Contact

Page Type

This page is about a company.

Retrieved from "http://aboutus.com/index.php?title=SitePerformanceMonitoringTools&oldid=11821770"