Wednesday, January 27, 2016

iperf3.. an old friend all shiny and new

So.. for a long time iperf has been a great friend..

You know what it's like when a friend you haven't seen in 18 months shows up for lunch 60 pounds lighter and wearing some snazzy threads you can't help but admire?

That's what iperf3 is like..

Installing via RH EPEL repo on CentOS 6 X86_64 as of today should end up looking like this if you install the optional devel package as well.

I used something like this:

"yum -y install iperf3 iperf3-devel"

Afterwards, "yum list installed | grep iperf" should produce output that looks kind of like this:

iperf3.x86_64           3.0.11-1.el6    @epel
iperf3-devel.x86_64     3.0.11-1.el6    @epel

So.. a little background..

1)  Some of us have multi-homed servers.  I personally wouldn't do anything as stupid as round robin default routing, but I might have some specific persistent static routes, especially useful since I cut a vlan tied directly into the core network, to bypass any sort of packet filtering devices (read: Firewalls) in my efforts to test a circuit.

2)  Static routes on and off a box I supposed I could replace by running Quagga and iBGP but that's over the top for a host that isn't supposed to perform a routing role in the network.

3)  I use a single physical interface on a kvm host in dot1q trunking mode, so to the kvm host, the interfaces are configured as "ethx.", and then those are bound to bridge interfaces on the kvm host.  This is the easy way to maintain as much or as little isolation between the different hosts on the system as well.  If the layer 2 traffic doesn't go there, it can't ever be pcap'd there ;)

Anyway.. back to iperf.

Really simple usage to drop it in server mode on the target host:

"iperf3 -s -fM -V --bind"

Really simple usage to fire up the client, bind to a specific address, pointed to the server and run the test, which in this case is specified to max at 5Mb/s of UDP trafffic, and run for 30 seconds.

"iperf3 -c --bind -b5M -u -V -t30"

Simple excerpt from "netstat -rn" verifying routing is going through the network elements I want it to from the kvm guest's second bridged interface (eth1) UG        0 0          0 eth1

And the end result is pretty dang nice.  I've tested it on TCP successfully at 100Mb, UDP, up to 120 so far.  It will take quite a bit to be able to test at line rate in terms of perhaps dedicated hardware and the willingness to dump 1Gb or 10Gb of traffic on a network, which is a bit more aggressive, but all in all it's pretty nice.

Aside even from PTP (VPL/EVPL/PL, or LAN) circuits, another great use is testing things like IPSEC throughput on a L2L connection.

Great stuff.  What's not to like?

Output from a test run (-R reverses the transfer direction for qualifying transmit and receive paths which can be assymetrical).

[root@iperf network-scripts]# iperf3 -c --bind -b5M -u -V -t30
iperf 3.0.11
Linux 2.6.32-573.7.1.el6.x86_64 #1 SMP Tue Sep 22 22:00:00 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
Time: Wed, 27 Jan 2016 18:09:59 GMT
Connecting to host, port 5201
[  4] local port 35963 connected to port 5201
Starting Test: protocol: UDP, 1 streams, 8192 byte blocks, omitting 0 seconds, 30 second test
[ ID] Interval           Transfer     Bandwidth       Total Datagrams
[  4]   0.00-1.00   sec   552 KBytes  4.52 Mbits/sec  69
[  4]   1.00-2.00   sec   608 KBytes  4.98 Mbits/sec  76
[  4]   2.00-3.00   sec   616 KBytes  5.05 Mbits/sec  77
[  4]   3.00-4.00   sec   608 KBytes  4.98 Mbits/sec  76
[  4]   4.00-5.00   sec   608 KBytes  4.98 Mbits/sec  76
[  4]   5.00-6.00   sec   616 KBytes  5.05 Mbits/sec  77
[  4]   6.00-7.00   sec   608 KBytes  4.98 Mbits/sec  76
[  4]   7.00-8.00   sec   608 KBytes  4.98 Mbits/sec  76
[  4]   8.00-9.00   sec   616 KBytes  5.05 Mbits/sec  77
[  4]   9.00-10.00  sec   608 KBytes  4.98 Mbits/sec  76
[  4]  10.00-11.00  sec   608 KBytes  4.98 Mbits/sec  76
[  4]  11.00-12.00  sec   608 KBytes  4.98 Mbits/sec  76
[  4]  12.00-13.00  sec   616 KBytes  5.05 Mbits/sec  77
[  4]  13.00-14.00  sec   608 KBytes  4.98 Mbits/sec  76
[  4]  14.00-15.00  sec   608 KBytes  4.98 Mbits/sec  76
[  4]  15.00-16.00  sec   616 KBytes  5.05 Mbits/sec  77
[  4]  16.00-17.00  sec   608 KBytes  4.98 Mbits/sec  76
[  4]  17.00-18.00  sec   608 KBytes  4.98 Mbits/sec  76
[  4]  18.00-19.00  sec   608 KBytes  4.98 Mbits/sec  76
[  4]  19.00-20.00  sec   616 KBytes  5.05 Mbits/sec  77
[  4]  20.00-21.00  sec   608 KBytes  4.98 Mbits/sec  76
[  4]  21.00-22.00  sec   608 KBytes  4.98 Mbits/sec  76
[  4]  22.00-23.00  sec   616 KBytes  5.05 Mbits/sec  77
[  4]  23.00-24.00  sec   608 KBytes  4.98 Mbits/sec  76
[  4]  24.00-25.00  sec   608 KBytes  4.98 Mbits/sec  76
[  4]  25.00-26.00  sec   616 KBytes  5.05 Mbits/sec  77
[  4]  26.00-27.00  sec   608 KBytes  4.98 Mbits/sec  76
[  4]  27.00-28.00  sec   608 KBytes  4.98 Mbits/sec  76
[  4]  28.00-29.00  sec   608 KBytes  4.98 Mbits/sec  76
[  4]  29.00-30.00  sec   616 KBytes  5.05 Mbits/sec  77
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  4]   0.00-30.00  sec  17.8 MBytes  4.99 Mbits/sec  0.248 ms  5/2282 (0.22%)
[  4] Sent 2282 datagrams
CPU Utilization: local/sender 0.2% (0.0%u/0.2%s), remote/receiver 0.0% (0.0%u/0.0%s)

A few lost packets doesn't look good.. that's something I get to explore and validate where it's *not* occurring.

Hmm.. might be system tuning and virtio under kvm.

Giving credit where credit is due, I found this other link (of course) the day after I wrote this post which does a great job of comparing the old iperf and the new iperf3, and while they are talking about a specific vendor, all the iperf3 and linux stuff works on any linux, and very similar variables are also present in FreeBSD.  OpenSolaris is a bit different, but has most of these tuning options with different defaults.

All in all, very good with a different set of examples.

For most of us using Linux not at the bleeding edge of network throughput, let's say 200Mb/s or less, it's not likely to need those kernel tuning options but if you're qualifying links, be they LAN or WAN, iperf3 is ridiculously useful now.

Monday, October 19, 2015

CFEngine 3 basic master server migration method:

The usual advice is:
- bring up a new server that contains (in masterfiles) a copy of your site policy
- change your process for initializing new hosts to use the new server
- rebootstrap your existing hosts to the new server

This was provided by a friend who is extensively involved with 
CFEngine, great stuff I couldn't seem to google so easily before.


Tuesday, March 10, 2015

Medicine in America and thinking about others.

The current state of the medical care in America is simply maddening.

So, we have the ACA.. among it's many components, there's apparently a requirement now for insurance companies to have the social security number of every person who is insured, now that healthcare is seemingly inextricably tied to taxes, so when adding my newborn, I get asked for his SS#, even though he won't have one until they mail us back the card in the mail.

Here's another gigantic gripe.

Health Care practitioners in order to get better rates for malpractice insurance, require you to sign away ALL rights, and agree that any case be settled in the arbitration framework of their choosing.

Sign it with the annotation of "under duress" and listen to the condescension coming from the doctor and other folks.  I have never been involved in a medical malpractice case of any type, frivolous or
otherwise, but why would I sign away my legal all legal rights?

Since when is signing away ALL legal rights considered a way we're supposed to do business in America?

Now, everything I know about contract law states that a good contract protects both (or all if more than two) equally.  Why is it every single contract we run across is so one sided?

Literally one sheet of paper, out of a stack of 10 for a new patient (who's brother is already registered in the office) is about the patient and any patient specific information.  The other 5 forms are legal documents stating that you accept full responsibility for all charges, independent of what the insurance company does or is supposed to do (this one is very popular of late), other somewhyat redundant affidavits of personal liability, and then the one that shows in red on the bottom:

"By signing this document you waive all rights for any medical malpractice issues that might occur outside of our specific arbitration."

This is unfortunately a paraphrase, since they don't bother to pass off a copy of the triplicate NCR form.  Apparently 1 copy is for the doctor's office, 1 copy for the insurance company's records, and 1 copy for the insurance company's counsel.

What kind of a world is it we live in where the basic legal rights (to sue someone should they injure you) available to everyone as mandated by law, are precluded as a part of the the "Normal Course of Doing Business".

Another one I like to point out is the habit some offices have of signing a "Patient Privacy" document that states they can at any time, re-sell your PMI to any third party (assuming to make side cash for populating their demographics databases ), despite the HIPPA regulations prohibiting them from re-distributing or accidentally leaking your information.  By signing it and by the practitioner (I saw it at a chiropractor's office) including as part of the new patient paperwork, you give them the legal release to sell it repeatedly.  I refused to sign it and was told it was to protect me, when it clearly stated that is was asking me to absolve myself of all rights to privacy.

The most common response I've heard blindly parroted from practitioners and their staff when asked about having to sign financial liability or rights waiver paperwork is "It's for your protection."

It simply is not the case.  It's for the practitioners business and financial protection.  There isn't a single benefit to the consumer in any of the paperwork.  Legal regulations are one thing, and malpractice insurance conditions to maintain the rates they want to pay are completely another.

When I have insurance that mandates a specific co-pay and little to no out of pocket for the normal services, and little eligible for deductible, why would I have to sign a document stating I am completely 100% liable?  Trust is a two-way street, and it begins with the paperwork I get shoved in my face at the first visit.

Interestingly enough, doctor's in lower-income areas seem to be more demanding about this stuff than in higher income areas, understandably, they may run more risk of non-payment, but there has to be a better way than the "over-the-top" methods used today.  My old pediatrician's office was in 90278, the new one is in 93036.  The median income in those areas has a disparity of ~$33,000.

And maybe that's part of the problem.  As a society, we treat those who have the least, the worst, and those who have the most the best.  As much as I might be entrenched in looking after myself and my family, does that have to be to the detriment of everyone I may deem beneath me?  In theory no, in practice all over both this country and the world at large yes.

As with all things, I can only strive to be better myself, and to attempt to teach my kids to be even better than I am.  In fact, any real movement of change must be made up of it's component parts, which are all of us.

Remember that the true measure of "class" isn't how you treat those 'better' than you, but how you treat those beneath you.  Professionally, personally, in every single interaction.

Remember also, that people aren't measured on how well they handle easy situations, but difficult ones.  It's not easy to treat the little guys right, because theirs no likely direct benefit to you.

Think about that the next time you berate a Starbuck's employee for forgetting to put soy milk in your over-complicated latte' order.  Think about that when you see your mailman showing up a few minutes late for your package.

Remember that it's not about what you say, but what you do.  Make an effort to do something nice for someone, or treat someone better than they expect, and see what happens.  Once a day, once a week, once a year, all that really matters is the effort and the focus on the betterment of someone other than yourself.

Maybe my perspective on the medical profession is too selfish, but I stand by the basic premise of contract law, which states that any good contract protects both parties equally.

Friday, February 22, 2013

Open Source RC

So, there's a new (to me) thing about building diy scratch built foam aircraft..

A number of sites I could reference, especially, have a ton of information, but I digress, time to make it personal.

I, like many occasional hobby enthusiasts, bought an aircraft on sale and bound it to my transmitter, try 1, 10' and nose into the dirt after the rollover.  Try 2, maybe about 2' further, and since I gave it extra throttle, it's now busted.  There goes $65 goodbye, or is it?

The donor aircraft was a ParkZone F27Q Stryker 180, it's a 180 3000 rpm 2s brushless rig, that can push the stryker up to about 60mph..

However, with the right fan on the front of it, it can also get a whole new lease on life, being pretty lightly used..

In comes rc groups.. some guy who goes by the name "MIT KID" posts some plans, in CAM format, for cutting 3mm depron foam into a few patterns.. add a little CA glue, some tape, maybe some plastic wrap, and you've got a home built 'foamy' 3d aircraft.. huge control surfaces, 15mph top speed with a little junker motor, not really even particulary aerodynamic, but did I mention huge control surfaces?

Loosely patterned after the famous "Yak 54", it costs $22.50, including shipping from a guy who basically just cuts them for people.

Larger ones for around $40 can be had and you add a 280 brushless motor and esc/servos and do tail stands all day..

Crash 'em?  They weigh jack, pretty hard to break.  Heaviest thing on the plane is the battery, and your primary CG placement change.  If they do break, foam-safe CA glue will literally bond anything in the aircraft, and it still flies.

The Stryker motor needs to get converted to a puller, instead of a pusher by changing two wires, and the propeller gets changed for a proper puller prop that's larger and it's a whole new plane.  The motor mount for the stryker can be trimmed with a dremel pretty easily and made to mate right up to the front.

Similar ones in the 280 size from Parkzone sell for $200.  The Stryker is now on sale for $65.  Great investment if you count the re-incarnations possible.

Thursday, September 20, 2012

The Power of Unix or "Can it make me a PB&J?"

So, I was talking to a friend, and I was mentioning how powerful Unix is, and all these various things it can do, and he made the mistake of challenging whether or not it could make a PB&J. I think it's worthy of an comic, but I'm not the author unfortunately (the author is beyond brilliant!). In lieu of authorship of xkcd, I offer this exchange to the masses.

9:51 Friend:
Can it make my pb&j sandwich or do I have to go downstairs and do it?

9:52 Me:
It absolutely could.
No problem.
You wouldn't even have to go downstairs..
But you'd have to build the mechanisms. and then wire serial into the servo's.

9:53 Friend:
Ah you got me

9:53 Me:
I'd set it up with two automated squeeze dispensers.
Basically two mechanical hands with rubber covered fingers. gripping plastic squeeze bottles. and you'd want a warmer on the hand holding the PB so it could squeeze a little easier..
I'd say about 88 degrees F would probably do.. although 98-108 would make it run nice.
Then an elevator with an attached plate. and bread loaded into the bread hopper.
Which would have to slide the bread off slice by slice and lay it over on the plate..

9:55 Friend:
Ok it was meant as ajoke, now you've taken it beynod hypothetical to conditional reality... I gotta make a sandwich

9:55 Me:
a simple mechanical device.. I know.. I'm just saying.. it could.. and I could tell you how to do it.

9:55 Me:
it's just that f-ing powerful

Saturday, January 1, 2011

10 (or more) gripes about automotive-related businesses in America: Are you serious?

I'll start out by saying that I have a huge problem with the vast majority of people in any business relating to the sale, repair, and maintenance of automobiles.

Here's why:

1) Everyone in that industry assumes (often correctly) that you don't know anything about how a car works. Perhaps not everyone's interested in doing their own brakes, changing the oil, garage engine swaps, suspension upgrades, or even basic maintenance, or troubleshooting what's wrong with a vehicle, but knowing how and what is done can be gleaned without ever lifting a wrench. A read through the factory service manual (most cost somewhere around $100) can save you literally thousands of dollars worth of "bogus" charges and mechanic/shop owner's BS.

2) Most places tend to act like they are doctor's, and you wouldn't question your doctor's advice, would you? Most places invite you to get a second opinion, because you have a 1 in 100 chance of actually finding someone who knows precisely what's wrong with your car and is honest enough to tell you. In many cases, highly 'rated' shops are the worst offenders.

3) Consumers are under the mistaken impression that the dealer knows more about their car than anyone else, and that the dealer mechanics are more highly trained. While in some cases, that is true, the vast majority of dealer mechanics go to an ASE tech school for maybe a year and get really good training. That doesn't mean they remember any of it, or know how to call it up when necessary. The vast majority of dealer mechanics apprentice at the dealership doing tire rotations and oil changes (Usually making not much over minimum wage), and the dealer charges can shoot as high as $150 an hour, for a $10/hr guy to change your oil, maybe not remembering to tighten the drain plug entirely (it has happened, literally).

4) Automotive manufacturers do nothing to dispel these myths about dealers. They are quite happy with the relationship, and will do absolutely anything to ensure continued sales of whatever it is they are selling, and generally doesn't regulate dealer selling price for parts,
dealers are more than able to charge 2-5 times list on common service parts (Case in point, my local BMW dealer charges $36 for 1 single Bosch twin-fire copper sparkplug. I can buy a full set of the exact sparkplug from my local auto parts store for $3.99 each.

The aforementioned fear and misbelief drives people to accept this usury, BMW OEM or OES parts are great, many aftermarket stock replacement parts are equally wonderful. BMW is by far not the only consumer abuser, Infiniti (Nissan) is also pretty bad these days.

5) Every step of the process, from the first time you walk on the lot, to the time (often 3-6 hours later) when you drive off with your car (or the contract and a scheduled pickup time) is designed to make you accept this usury. The most common dealer tactic is to prey on your pride first, then switch to being outright degrading. Most people who sell cars for a living are not the type of people you typically choose to do business with, drug addicts, ex-cons, anybody who can't find anything else.

It's feast or famine, if you sell cars, you can make pretty decent money (around 100-150k for a sales manager at a dealership, including overrides on every sale made while on-shift), or you can starve. Imagine most sales guys are pretty hungry.

From the time you walk on the lot, people who are master con-artists are sizing you up and deciding how to approach you. I wish people could see the side of these guys that comes out when no customers are around.

5) Local mechanic repair shops, smog stations, both test only and test/repair.

A typical consumer has about a 1 in 100 chance of finding an honest mechanic. The same is true with smog shops. I recently had an experience with a smog shop that is all too typical. Basically, if you walk into a shop knowing anything at all about cars, you get immediately lowered priority. Just like walking into a dealership, you are getting sized up the moment you walk in. If you seem like an easy mark, they are all friendly, the minute they get the sense they can't steal you blind, all of the sudden they are too busy to help you.

6) Most mechanics don't have a high school education. This is changing somewhat these days, as a high school diploma and possibly some sort of technical school is a requirement for many shops, but at least 10 times as many don't have any such requirements. Most mechanics start out working for someone else, and eventually hang out their own shingle as a proprietor. 5-10 years of experience is a good amount to be able to handle the majority of vehicular repairs.

7) Book rates. There's this book, or set of books, published annually, that lists the 'rates' for various repairs. This has nothing to do with time worked. I know a mechanic who bragged to me that he once billed 17 hours (@$85 per) for a single 8 hour day worked. The mechanic working for someone else, gets paid for the job at a reduced hourly rate for the hours defined in the 'book' (usually a CD/DVD or online software like Alldata).

For instance.. let's say o2 sensor replacement says .5 hours in the book. The mechanic does it in 10 minutes, including time to lift the car up for access to the exhaust system (although sometimes that's not even necessary, and they are visible when you look under the hood). The mechanic gets paid for a half an hour, say $12.50 ($25/hr for an experienced mechanic in many markets). The shop turns around and bills you 1/2 hour labor @ $85-125 per hour, everyone makes money.

Here's where it gets better. Book rate 'usury' is based on an established set of rates, by a third party. Many shops actually overcharge even the already excessive book rate. I just had a shop attempt to charge me 1.5 hours of labor for an o2 sensor. When I argued about it, knowing the 'book' rate, the labor charge got cut down to a more reasonable number. The mechanic literally tried to charge me labor time for driving to a mexican restaurant down the street, bringing lunch back, and eating it, while having a friendly 45-minute lunch, chatting me up the whole time.

8) Mechanics have to spend so much on their certifications and tools, that they have to resort to theft and usury just to make ends meet? I call BS on that one, straight up. I have bought a complete set of shop tools before, and other than the very expensive ones, made by companies like Snap-On and MAC tools, do cost a small fortune. Far more than they are worth. You know how these guys get hooked on them when Craftsman will more than do the job? Three ways,
first, they offer financing. A $20,000 roller tool chest/cabinet that's worth $50k after it's filled with overpriced tools, looks attractive when a guy can have it for $200 a month. They can't finance 5k worth of Craftsman tools like they can Snap-On. The second way is by offering things like a student discount. The third method is the same way the industry sells you on stuff, pride. Especially in Southern California, where people all over the place brag about how much they spent on whatever. Back on the east coast, most of the folks I know and love brag about how good of a deal they got, not how much they overpaid.

I love great tools as much as the next guy, but I can tell you that for $300, I can have 98% of the tools necessary to do just about anything on a car. For $1000, I could do a complete overhaul, sans machine work, including several pricey specialty items for at least one car make (which is common for most home automotive enthusiasts). Most mechanics pay off that $20-50k in tools within 3 years, many do it in less. How does that compare to the 3-500K in debt that a doctor incurs just for schooling and has to pay a huge malpractice insurance premium whether they ever have a claim against him/her or not? It doesn't. There are many other professions that incur a huge debt to get into. Aviation mechanics are required to have a lot more certification and tools.

9) Outright theft is rampant, and largely unnoticed, and then often not prosecuted. Back when I didn't ever touch my own car, many years ago, I had an extremely reputable shop charge me no less than $12k to fix a number of long-ignored, somewhat difficult maintenance items on a BMW 750iL that cost $7k originally. Had everything been done as paid for, I would have only called it my fault for accepting the deal. It was over the course of 3 visits at about 4k per visit, claiming to do some major maintenance work. It was a very expensive lesson, when the very same items got looked at by a shop 3 months later, and were found to be completely un-repaired. I will never forget that experience. My fiance took her still-under-warranty Honda to the local dealership for an oil change, against my advice (you can't always teach people by explanation, they have to see it for themselves) and got informed she needed new brake pads in the front, for $300.

I can tell you personally that for a mechanic with a car lift and air tools, after the 5 lugs for the wheels come off with an air impact wrench, There are two caliper pins that come off, and the brake pads slide out after removing the caliper from the bracket. I can do it without a lift in any driveway (it doesn't even have to be perfectly level) in about 30 minutes, both sides, including spreading the caliper with a large pair of channel-lock pliers.

The other 'special' tools you need for the job are a flat-head screwdriver and a basic wrench set (gear wrenches are a huge plus, and if you fork over $80 for a nice set, they will get used every time you open the hood). New pads on average run about $45-60, and that's if you don't buy them for $20 from a place like If you're going to replace 3 year old pads, you need to replace the rotors. 'Turning' or resurfacing the rotors takes metal away, and while it may provide a nice surface for your pad, you want your brake rotors to have as much metal as possible, but the full explanation of the engineering principles behind braking is a topic for a different conversation.

If you buy your rotors from the dealer, they will charge you a small fortune for machined hunks of cast steel. Always buy the cheapest rotors you can find, you will find that these hunks of steel are all basically equal. You don't get anything for your money buying expensive hunks of disposable steel. Dealers, auto-parts stores, and aftermarket vendors rely on people's ignorance of the physics behind braking and the purpose of brake rotors, to imply that paying $150 for a single brake rotor when you can buy one for $28 actually makes some sort of sense.

I check all my various parts stores when I need to buy anything, to figure out who has the best price, including shipping or tax (or both in rare cases), and I often find the shipping cost for brake rotors far exceeds the price including tax for locally picking them up. This isn't always the case, shop around.

10) Myth smithery at large: If you have a car, and you're 'enthusiastic' about understanding and working on it, you have a far greater vested interest in maintaining or fixing it properly, and at minimum cost. Ergo, you are going to purchase the tools you need to properly diagnose it, these days, that's usually a OBDI/II code reader of some type, and a $10-20 digital multi-meter. Between those two tools, you can test/diagnose most of the electrical components in a vehicle, and obtain the excellent information contained within the OBD (On-board diagnostics). Here's the problem, cars are complicated systems, and in many cases, the information isn't perfect, although if you have a code for an o2 sensor, replacing it generally clears the code (and increases your fuel milage by 10-30% depending on how bad it runs with a failed one). They last 60-80k miles usually, so replacing them by 80 or every 80k is very recommended.

11) New parts, used parts, salvage yards, and take off parts. What most mechanics do when you need something for your car is order it from his local auto-parts store or distributor, and the distributor delivers it for free in his Ford Ranger or similar tiny truck, and the mechanic marks it up 50-100% and writes down the new number on your bill. I don't know about everyone else, but I shop around. The old excuse is let the guy make a little on the parts, I can't buy it as cheap as he can. In general, the re-sale business is predicated on entities buying something for less, (usually in some volume) and selling it at a competitive markup. This is a tried and true business model, that works. For $50 on a $120 item, I'm happy to spend a little time shopping and order the parts I need. Most people don't go into a shop knowing what needs to be fixed, much less what it takes to fix it, or what parts to order.

Many parts can be purchased used but functional from salvage yards, or even better yet, although many of them have taken to charging 50% of retail for parts with 60-120K miles on them. You have to shop around there as well, just like anywhere else.

Take off parts are something different entirely. Let's say you go into a shop and ask them to install an aftermarket exhaust. They remove your stock one, install the aftermarket one, and unless you specifically ask, keep the old parts. Dealers are the worst this way. The mechanic then takes the money he made from overcharging you on labor (book rates are an overcharge, it's labor charged for labor not performed, and most mechanics who have been working for two years can do the majority of book rate jobs in half the time specified in the 'book'), and then take your perfectly functional, relatively expensive parts, and sell them on craigslist, ebay, or an enthusiast forum for some good undocumented cash. I recall talking to a mechanic at an Infiniti dealer who bragged about making over $25,000 in one year like that, and showed off his expensive toys to prove it.

Most people assume that if someone's does it for a living, they know more about it than you do, this is not really true. You know instantly if something's wrong with your car, you can feel it through the floorboards, or in the throttle/braking response, just having low pressure in one tire can cause the car to wander, forcing you to correct as you drive in a straight line. Looseness in the steering feel is due to worn tie rods, or other sacrificial suspension joints such as bushings, ball joints, etc. Most all of these are designed to be replaced relatively easily, and at a relatively minimum cost because they are WEAR items. Really, every single thing on a car is a wear item, but some much more so than others. The methods for checking these things including wheel bearings, largely revolves around wiggling the steering wheel around while someone else looks under the car to see where the joints have 'play'. This isn't rocket science, it's not even automotive engineering, and quite honestly is a lot like playing with an erector set and looking for the weak link. An untrained eye if looking at the joints closely, can see where the play is, and in most cases, visual inspection is the only diagnostic method available.

So, what's up with this industry? We are so reliant upon the automobile, and yet it is often our collective blind spot? Don't we all have a vested interest in making sure we're not getting shafted? Where's the value? Why is it if someone smiles and gives us a false sense of security, we accept financial buggery as a matter of course?

Let's look at this from an energy exchange perspective, favored by the 'Burners' (People who attend Burning Man).

I work at a decent salary, and I also do a certain amount of side work, for an hourly rate. Let's say that rate is $75 an hour. I also have a retainer job that adds a modest amount monthly, and doesn't always require attention, but that's special, and not usury by any stretch, I am generally relatively generous about billing for overages.

In an energy exchange, I put out a certain amount of energy. Usually for an hour billed, it's about an hour and 15 minutes of fairly intense mental concentration, using every tool at my disposal, and my not inconsiderable experience, gained from starting out 'building computers' 20 years ago, and being a unix sysadmin and adding quite a bit of networking/security skills over the years, all gained via mentors, self-teaching via O'Reilly books, classes, and search engines/forum and mailing list posts, etc.. The amount of energy behind that hour is immeasurable by me, and quite a bit more than that hour would indicate.

Now, I feel, that I offer a distinct value to my clients. I'm fast, experienced, and a 'pit-bull' when it comes to figuring out something that's stumping me. In order to justify myself to my own code of ethics, if I don't put out an hour's worth of work, I don't bill an hour. It makes sense to me and my clients.

Even more importantly, while it used to be that mechanics were relatively inexpensive, fair, and basically honest folks, those days are long, long, gone. Every hour I invest in my own knowledge and experience with my car is worth it. If it takes me 2 hours to do something that a shop could do in one hour, it's still a good investment. If I made $450 per hour, I would drop it off and pay the $125 an hour all day long, but when 2 hours labor is more than I make in a day at my considerably more complex and rare skill-set occupation in a day, than the energy exchange is off-balance. The entire monetary system is about equal exchange of energy.

Why is it that American business has the saying 'It's nothing personal, just business' as an excuse for lying, stealing, and generally screwing people in the name of the almighty dollar? Part of it, is that we don't have time to educate ourselves, or are afraid of the learning curve. Part of it is that we buy into the expense of having our own mobility, ego-fulfillment, and this perceived self-reliance that allows. Part of it is that cars are FUN to drive. There's something intensely satisfying about being able to hurtle through space much faster than any human could on any human-powered transportation. Why do people who don't have the skill or interest to drive at 170MPH properly on a racetrack buy cars that are designed to be able to? It's a blast! I totally get it.

Ludicrous maintenance costs are accepted by the American consumer, unless you buy a Toyota and for the high line brands, (anything that sells for over $50k new) it's a matter of course. Every car has it's special tricks and issues, and the occasional specialty tool, but all internal combustion engines needs similar maintenance, and have nearly identical diagnostic methods.

Why is it that in every other industry, we have an expectation that value received is in some way commensurate with dollars spent? Why do we accept usury, theft, and lying from automotive industry businesses and why is it we have laws to protect against that sort of stuff, but we're all still getting shafted? The answer is sloth, complacence, and the fact that it's largely a unified front. When I find a good mechanic (I've found 3 in my lifetime) I refer them all the business I can. I look for guys willing to bill me on hours worked, not book rates, and in the rare circumstances where I simply cannot deal with the job myself, I will take it to the professional, who provides what a professional should provide:

1) Value. I take it to the pro, because he can diagnose and resolve the issue in less time than it would take me to do it myself, and because he has the tools or diagnostic equipment to do the job cheaper, faster, and better than I can myself.

2) Honesty. In anybody I do business with for literally anything, honesty is a must. I control that by voting with my wallet every single chance I get. I will happily speak up loudly to call out those who do not do honest/fair business.

3) Experience. I've worked on 25 cars in my life. Your average mechanic sees that many in less than 2 weeks. 3-7 cars per day isn't uncommon for a good mechanic who works a little more than an 8 hour day.

The unfortunate reality is that none of these things are a given. When you go to a shop these days, mechanics tell you that you need to replace super expensive items, pulling these answers out of their asses, and most people go for it. A recent example is where I'm being told I need to change my catalytic converters on my 16.5 year old BMW, because the HC's are too high.

The reality is that the engine was running rich due to knock sensors being bad, and the ignition is incomplete because the coil packs are 16.5 years old and 154,000 miles old, and the combination of under-hood heat, vibration, and petroleum products takes it's toll on copper-wrapped ignition autotransformers (coils). A great article explaining ignition coils is here:

I tested the coils individually and found them either just at the upper limit of the acceptable range of impedance (.8 ohms) or over it. I bought a $20 digital multi-meter to test this, it took about 30 seconds for each one times 6 and about 5 minutes to open the hood, hook up the light, and remove the plastic cover from the top of the engine. BMW factory coils are over $80 a piece, without the boots to connect them to the spark plugs, but if you don't require a BMW part number stamped on them, FCP Groton sells a set of 6, including the boots for $150.

Obviously, this entry has taken on a life of it's own. I probably could have written no less than 10 separate articles, and it probably would have been a little easier to follow, but sometimes you just have to dump all the passion into one shot. I believe I've been typing for a little over 2 hours now, so it's time to close!


Monday, November 1, 2010

Stop with the "Core" in your name already..

It's been almost a year since my last entry. Boy time flies when you're building a telephone company!

Lately, I've been noticing some trends in the industry, that are really starting to bother me a bit.

1) Stop blowing money on a corporate image change and focus on better delivery of core services at lower prices and better margins.

No more "Core" in your name. It sounded cool when I was watching Beavis and Butthead.

Heh heh. He said "Core".

2) Stop with the managing of expectation, and take some pride in rolling up your sleeves and getting the job done.

Pay special attention to MTTR (Mean Time To Resolution), that is the most important factor.

When software functionality that's both obvious and critical gets ignored for 3 years, or routine support issues get dragged out for weeks because the person who's capable of dealing with it is "busy" or an outside vendor, people are understandably upset.

Pretty much the only valid response for a support organization is one of a triage surgeon - what can I do to stop the bleeding and ensure this type of injury never happens again?

Oh yeah, everyone's in the support organization. Sales, product, customer service, support, even the janitor.

It's just as true for the emergency room as it is for MSP's, ISP's, Telecom, in house IT organizations, and just about anybody in a support role in any industry.

3) There is nothing worse than someone non-technical who can't do anything useful for the problem to spout off about it incessantly, only to backpedal later and act sheepish. If you're out of your depth, or can't bring anything useful to the conversation, less is more - bow out gracefully.

4) Escalate. If someone's asking for escalation, they pretty much mean it. Any support request more than 2 days old is perceived by the customer as a "bad" support incident. Conversely, almost any response in less than 2 hours is considered "good", even if it's just a request for clarification from Tier 1.

5) Clarification. The engineer needs to be absolutely certain that the issue as they know is the issue. Don't trust any source but the customer. There's nothing worse than blowing an hour trying to troubleshoot an unclear issue. Conversely, reward complete information with the speediest possible service. When there's an opportunity for dialog leading to resolution, get on the phone faster with less information and get it straight from the horses mouth. This is where those hacker-style social engineering skills really pay off. NLP for dummies is also a strongly recommended read, to help you develop a rapport as quickly as possible with the customer.

6) Callback sooner rather than later. In my Tier 3 support role, I will often get up early and make the first few calls over coffee. Nothing says "Your issue is important to us." like a phone call. There is nothing that makes someone happier than having their issue picked up by someone who can actually do something about it.

7) Process is important in a support organization, but when it gets in the way of support to the degree that issues sit around for a week or two, there's a problem with the process. If your organization is in between 5-200 people, you probably have some process getting in the way of productivity. Make every effort to analyze and reduce all process overhead to as little as possible. Better living through automation and not replicating derived information.

8) When modeling your support processes, take cues from the big guys, but don't change your hold music to match them, and don't institute the process, if you don't have the man-hours to fulfill it. This one is the most often overlooked. When you're a 6-person shop, everyone should have instant messaging and pretty easy multi-line phone access. When you're a 50 and larger headcount, communication becomes more important than anything else. I strongly recommend using Google docs and a wiki internally for keeping information that everyone involved can edit and maintain. The key with this is maintenance, if you find something wrong in the documentation, you fix it and move on.

9) Accountability and Transparency.

What happened to "The buck stops here!"? What about "The customer is always right.", or commitments to timeline that actually mean something? There's quite a few businesses out there these days that are floundering because of a corporate attitude, not a lack in talent or resources. Transparency means, whatever the problem is, we empirically prove it out and address it so it can't happen again to everyone's satisfaction..

Companies I want to highlight who do an incredible job of transparency and accountability:


10) High level review.

Management at every level needs to be in touch with the customer experience. I cannot emphasize enough how important that is. Escalation needs to be able to go all the way up the food chain to the top if necessary. Even in a 500-person organization, the attitude of being willing to do whatever it takes to make it happen needs to be re-enforced everywhere.