Saturday, January 30, 2016

salva discos 2

So your reshape/grow crashed for some reason. mdadm says this:

# mdadm -A --scan
mdadm: Failed to restore critical section for reshape, sorry.
      Possibly you needed to specify the --backup-file

# mdadm -A --scan --verbose
mdadm: /dev/md/5 is identified as a member of /dev/md/3, slot 3.
mdadm: /dev/md/4 is identified as a member of /dev/md/3, slot 2.
mdadm: /dev/md/1 is identified as a member of /dev/md/3, slot 0.
mdadm: /dev/md/2 is identified as a member of /dev/md/3, slot 1.
mdadm:/dev/md/3 has an active reshape - checking if critical section needs to be restored
mdadm: Failed to find backup of critical section
mdadm: Failed to restore critical section for reshape, sorry.
      Possibly you needed to specify the --backup-file

Limit the damage

First step is to limit the damage. We may have to experiment a little and we really do not want to restore a backup just because we changed a few MB. You can overlay a device with a file: Writes will go to the overlay file, and reads will try the overlay file first and then the actual device. The files can be sparse files and thus will only take up as much space as is written to them.

## Set the devices we want to overlay 
DEVICES="/dev/md/1 /dev/md/2 /dev/md/4 /dev/md/5"
## Create a /dev/loop for each of the files
parallel 'test -e /dev/loop{#} || mknod -m 660 /dev/loop{#} b 7 {#}' ::: $DEVICES
## Create some sparse files. If 4000G is too big, lower that
parallel truncate -s4000G overlay-{/} ::: $DEVICES
## Setup the overlay 
parallel 'size=$(blockdev --getsize {}); loop=$(losetup -f --show -- overlay-{/}); echo 0 $size snapshot {} $loop P 8 | dmsetup create {/}' ::: $DEVICES
## Make a variable with the overlay devices
OVERLAYS=$(parallel echo /dev/mapper/{/} ::: $DEVICES)
## Print them 
echo $OVERLAYS
## Check the status on the overlay devices
dmsetup status

You will later undo the overlay files with:

## Dont do this now.
parallel 'dmsetup remove {/}; rm overlay-{/}' ::: $DEVICES
parallel losetup -d ::: /dev/loop[0-9]*

Check you get the same error:

# mdadm -A /dev/md3 $OVERLAYS
mdadm: Failed to restore critical section for reshape, sorry.
      Possibly you needed to specify the --backup-file

 Check how far the reshape got (Reshape pos'n):

# mdadm -E $OVERLAYS
/dev/mapper/1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x4
     Array UUID : 7529d9f4:3e69776c:df6bf129:ffd1f902
           Name : lemaitre:3  (local to host lemaitre)
  Creation Time : Mon Nov  5 17:52:58 2012
     Raid Level : raid4
   Raid Devices : 5

 Avail Dev Size : 54698242800 (26082.16 GiB 28005.50 GB)
     Array Size : 109396484096 (104328.62 GiB 112022.00 GB)
  Used Dev Size : 54698242048 (26082.15 GiB 28005.50 GB)
    Data Offset : 16 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : f4d73202:5d6af3b9:c13d9a84:ef67abc0

  Reshape pos'n : 109169573888 (104112.22 GiB 111789.64 GB)

    Update Time : Wed Jul 10 09:51:37 2013
       Checksum : f16a7a58 - correct
         Events : 15431828

     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AAAA. ('A' == active, '.' == missing)

Assembling still does not work with --force and --run:

# mdadm --assemble --force --run --verbose $OVERLAYS
mdadm: device /dev/mapper/1 exists but is not an md array.
root@lemaitre:/lemaitre-internal# mdadm --assemble --force --run --verbose /dev/md3 $OVERLAYS
mdadm: looking for devices for /dev/md3
mdadm: /dev/mapper/1 is identified as a member of /dev/md3, slot 0.
mdadm: /dev/mapper/2 is identified as a member of /dev/md3, slot 1.
mdadm: /dev/mapper/4 is identified as a member of /dev/md3, slot 2.
mdadm: /dev/mapper/5 is identified as a member of /dev/md3, slot 3.
mdadm:/dev/md3 has an active reshape - checking if critical section needs to be restored
mdadm: Failed to find backup of critical section
mdadm: Failed to restore critical section for reshape, sorry.
      Possibly you needed to specify the --backup-file

Try --invalid-backup

Next step is try to assemble without the backup file:

# mdadm --assemble --verbose --invalid-backup --force /dev/md3 $OVERLAYS
mdadm: looking for devices for /dev/md3
mdadm: /dev/mapper/1 is identified as a member of /dev/md3, slot 0.
mdadm: /dev/mapper/2 is identified as a member of /dev/md3, slot 1.
mdadm: /dev/mapper/4 is identified as a member of /dev/md3, slot 2.
mdadm: /dev/mapper/5 is identified as a member of /dev/md3, slot 3.
mdadm:/dev/md3 has an active reshape - checking if critical section needs to be restored
mdadm: Failed to find backup of critical section
mdadm: continuing without restoring backup
mdadm: added /dev/mapper/2 to /dev/md3 as 1
mdadm: added /dev/mapper/4 to /dev/md3 as 2
mdadm: added /dev/mapper/5 to /dev/md3 as 3
mdadm: no uptodate device for slot 4 of /dev/md3
mdadm: added /dev/mapper/1 to /dev/md3 as 0
mdadm: array: Cannot grow - need backup-file
mdadm: failed to RUN_ARRAY /dev/md3: No such file or directory

# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md3 : active (read-only) raid4 dm-0[0] dm-3[3] dm-2[4] dm-1[1]
      109396484096 blocks super 1.2 level 4, 512k chunk, algorithm 5 [5/4] [UUUU_]
   
Looks good but is read-only. Make it read-write with:

# mdadm --readwrite /dev/md3

# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md3 : active raid4 dm-0[0] dm-3[3] dm-2[4] dm-1[1]
      109396484096 blocks super 1.2 level 4, 512k chunk, algorithm 5 [5/4] [UUUU_]
      [===================>.]  reshape = 99.7% (27294226944/27349121024) finish=30.4min speed=30013K/sec
 
Much better. But the reshape is going into the overlay files, so we may run out of disk space. Slow down the reshape for now:

# echo 0 > /proc/sys/dev/raid/speed_limit_max
# echo 0 > /proc/sys/dev/raid/speed_limit_min
# sleep 30
# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md3 : active raid4 dm-0[0] dm-3[3] dm-2[4] dm-1[1]
      109396484096 blocks super 1.2 level 4, 512k chunk, algorithm 5 [5/4] [UUUU_]
      [===================>.]  reshape = 99.8% (27295675392/27349121024) finish=1670176.0min speed=0K/sec

Great. Now let us assess the damage.

fsck

See what fsck says:

fsck /dev/md3 

If the file system uses xfs:

xfs_repair /dev/md3

Because we are doing this on overlay files there is no need to do a read-only fsck.

mount

Assuming the fsck completed and fixed any errors, mount the file system.

mkdir /mnt/disk
mount /dev/md3 /mnt/disk

XFS sometimes like to get unmounted again before first use, so:

umount /dev/md3
mount /dev/md3 /mnt/disk

Look around in /mnt/disk. Check /mnt/disk/lost+found:

find /mnt/disk/lost+found

If there are no files there, fsck did not rescue any files. That is typically a good sign, as that can mean that no directories were corrupt. Now check if the disk usage is what you expect:

# df /mnt/disk
Filesystem        1K-blocks        Used   Available Use% Mounted on
/dev/md3       109394397184 48950231280 60444165904  45% /mnt/disk
That looks good, I expected around 45% free, so if anything got lost it would be small things. And if fsck did not complain at all, then nothing was lost.

Doing it for real

Now it is time to disable the overlay files and do the same for real. You should have taken notes of the exact steps that worked for you earlier. If you did not, remove and add the overlay and do it again.

umount /mnt/disk
mdadm --stop /dev/md3
# Remove overlay
parallel 'dmsetup remove {/}; rm overlay-{/}' ::: $DEVICES
parallel losetup -d ::: /dev/loop[0-9]*

# Re-do the steps that worked for you
# For me it was:

DEVICES="/dev/md/1 /dev/md/2 /dev/md/4 /dev/md/5"
mdadm --assemble --verbose --invalid-backup --force /dev/md3 $DEVICES
cat /proc/mdstat
mdadm --readwrite /dev/md3
echo 0 > /proc/sys/dev/raid/speed_limit_max
echo 0 > /proc/sys/dev/raid/speed_limit_min
sleep 30
cat /proc/mdstat
xfs_repair /dev/md3
mkdir /mnt/disk
mount /dev/md3 /mnt/disk
umount /dev/md3
mount /dev/md3 /mnt/disk
find /mnt/disk/lost+found
df /mnt/disk
umount /dev/md3

Now let the reshape complete:

echo 30000 > /proc/sys/dev/raid/speed_limit_max
echo 30000 > /proc/sys/dev/raid/speed_limit_min

S
Posted by at No comments:

Salva discos!!

Recovering a failed software RAID

Notice: The pages "RAID Recovery" and "Recovering a failed software_RAID" both cover this topic. "Recovering a failed software_RAID" is safe to do as it does not make any changes to the RAID - except in the final stage.
The software RAID in Linux is well tested, but even with well tested software, RAID can fail.
In the following it is assumed that you have a software RAID where a disk more than the redundancy has failed.
So your /proc/mdstats looks something like this:
 md0 : active raid6 sdn1[6](S) sdm1[5] sdk1[3](F) sdj1[2] sdh1[1](F) sdg1[0](F)
     305664 blocks super 1.2 level 6, 512k chunk, algorithm 2 [5/2] [__U_U]
Here is a RAID6 that has lost 3 harddisks.
Before you try this document on real data, you might want to try it out on a bunch of USB-sticks. This will familiarize you with the procedure without any risk of losing data.

Contents

 [hide] 

Setting the scene

This article will deal with the following case. It starts out as a perfect RAID6 (state 1):
 md0 : active raid6 sdn1[6](S) sdm1[5] sdk1[3] sdj1[2] sdh1[1] sdg1[0]
     305664 blocks super 1.2 level 6, 512k chunk, algorithm 2 [5/5] [UUUUU]
For some unknown reason /dev/sdk1 fails and rebuild starts on the spare /dev/sdn1 (state 2):
 md0 : active raid6 sdn1[6] sdm1[5] sdk1[3](F) sdj1[2] sdh1[1] sdg1[0]
     305664 blocks super 1.2 level 6, 512k chunk, algorithm 2 [5/4] [UUU_U]
     [===>.................]  recovery = 16.0% (16744/101888) finish=1.7min speed=797K/sec
During the rebuild /dev/sdg1 fails, too. Now all redundancy is lost, and losing another data disk will fail the RAID. The rebuild on /dev/sdn1 continues (state 3):
 md0 : active raid6 sdn1[6] sdm1[5] sdk1[3](F) sdj1[2] sdh1[1] sdg1[0](F)
     305664 blocks super 1.2 level 6, 512k chunk, algorithm 2 [5/3] [_UU_U]
     [===========>.........]  recovery = 59.0% (60900/101888) finish=0.6min speed=1018K/sec
Before the rebuild finishes, yet another data harddisk (/dev/sdh1) fails, thus failing the RAID. The rebuild on /dev/sdn1 cannot continue, so /dev/sdn1 reverts to its status as spare (state 4):
 md0 : active raid6 sdn1[6](S) sdm1[5] sdk1[3](F) sdj1[2] sdh1[1](F) sdg1[0](F)
     305664 blocks super 1.2 level 6, 512k chunk, algorithm 2 [5/2] [__U_U]
This is the situation we are going to recover from. The goal is to get back to state 3 with minimal data loss.

Tools

We will be using the following tools:
GNU Parallel - http://www.gnu.org/software/parallel/ If it is not packaged for your distribution install by:
 wget -O - pi.dk/3 | bash

Identifying the RAID

We will need the UUID of the array to identify the harddisks. This is especially important if you have multiple RAIDs connected to the system. Take the UUID from one of the non-failed harddisks (here /dev/sdj1):
 $ UUID=$(mdadm -E /dev/sdj1|perl -ne '/Array UUID : (\S+)/ and print $1')
 $ echo $UUID
 ef1de98a:35abe6d9:bcfa355a:d30dfc24
The failed harddisks are right now kicked off by the kernel and not visible anymore, so you need to make the kernel re-discover the devices. That can be done by re-seating the harddisks (if they are hotswap) or by rebooting. After the re-seating/rebooting the failed harddisks will often be given different device names.
We use the $UUID to identify the new device names:
 $ DEVICES=$(cat /proc/partitions | parallel --tagstring {5} --colsep ' +' mdadm -E /dev/{5} |grep $UUID | parallel --colsep '\t' echo /dev/{1})
 {5}     mdadm: cannot open /dev/{5}: No such file or directory
 sda1    mdadm: No md superblock detected on /dev/sda1.
 sdb1    mdadm: No md superblock detected on /dev/sdb1.
 $ echo $DEVICES
 /dev/sdj1 /dev/sdm1 /dev/sdn1 /dev/sdo1 /dev/sdp1 /dev/sdq1

Stop the RAID

You should now stop the RAID as that may otherwise cause problems later on:
 mdadm --stop /dev/md0
If you cannot stop the RAID (due to the RAID being mounted), note down the RAID UUID and re-seat all the harddisks used by the RAID or reboot. Afterwards identify the devices again like we did before:
 $ UUID=ef1de98a:35abe6d9:bcfa355a:d30dfc24
 $ DEVICES=$(cat /proc/partitions | parallel --tagstring {5} --colsep ' +' mdadm -E /dev/{5} |grep $UUID | parallel --colsep '\t' echo /dev/{1})
 $ echo $DEVICES
 /dev/sdq1 /dev/sds1 /dev/sdt1 /dev/sdu1 /dev/sdv1 /dev/sdw1

Check your hardware

Harddisks fall off a RAID for all sorts of reasons. Some of them are intermittent, so first we need to check if the harddisks are OK.
We do that by reading every sector on every harddisk in th RAID.
 parallel -j0 dd if={} of=/dev/null bs=1M ::: $DEVICES
This can take a long time (days on big harddisks). You can, however, leave this running while continuing through this guide.

Hardware error

If the reading fails for a harddisk, you need to copy that harddisk to a new harddisk. Do that using GNU ddrescue. ddrescue can read forwards (fast) and backwards (slow). This is useful since you can sometimes only read a sector if you read it from "the other side". By giving ddrescue a log-file it will skip the parts that have already been copied successfully. Thereby it is OK to reboot your system, if the copying makes the system hang: The copying will continue where it left off.
 ddrescue -r 3 /dev/old /dev/new my_log
 ddrescue -R -r 3 /dev/old /dev/new my_log
where /dev/old is the harddisk with errors and /dev/new is the new empty harddisk.
Re-test that you can now read all sectors from /dev/new using 'dd', and remove /dev/old from the system. Then recompute $DEVICES to include the /dev/new:
 UUID=$(mdadm -E /dev/sdj1|perl -ne '/Array UUID : (\S+)/ and print $1')
 DEVICES=$(cat /proc/partitions | parallel --tagstring {5} --colsep ' +' mdadm -E /dev/{5} |grep $UUID | parallel --colsep '\t' echo /dev/{1})

Making the harddisks read-only using an overlay file

When trying to fix a broken RAID we may cause more damage, so we need a way to revert to the current situation. One way is to make a full harddisk-to-harddisk image of every harddisk. This is slow and requires a full set of empty harddisks which may be expensive.
A faster solution is to overlay every device with a file. All changes will be written to the file and the actual device is untouched. We need to make sure the file is big enough to hold all changes, but 'fsck' normally will not change a lot, so your local file system should be able to hold around 1% of used space in the RAID. If your filesystem supports big, sparse files, you can simply make a sparse overlay file for each harddisk the same size as the harddisk.
Each overlay file will need a loop-device, so create that:
 parallel 'test -e /dev/loop{#} || mknod -m 660 /dev/loop{#} b 7 {#}' ::: $DEVICES
Now create an overlay file for each device. Here it is assumed that your filsystem supports big, sparse files and the harddisks are 4TB. If it fails create a smaller file (usually 1% of the harddisk capacity is sufficient):
 parallel truncate -s4000G overlay-{/} ::: $DEVICES
Setup the loop-device and the overlay device:
 parallel 'size=$(blockdev --getsize {}); loop=$(losetup -f --show -- overlay-{/}); echo 0 $size snapshot {} $loop P 8 | dmsetup create {/}' ::: $DEVICES
Now the overlay devices are in /dev/mapper/*:
 $ OVERLAYS=$(parallel echo /dev/mapper/{/} ::: $DEVICES)
 $ echo $OVERLAYS 
 /dev/mapper/sds1 /dev/mapper/sdt1 /dev/mapper/sdq1 /dev/mapper/sdu1 /dev/mapper/sdv1 /dev/mapper/sdw1
You can check the disk usage of the overlay files using:
 dmsetup status

Reset overlay file

You may later need to reset to go back to the original situation. You do that by:
 parallel 'dmsetup remove {/}; rm overlay-{/}' ::: $DEVICES 
 parallel losetup -d ::: /dev/loop[0-9]*

Overlay manipulation functions

devices="/dev/sda /dev/sdb /dev/sdc"

overlay_create()
{
        free=$((`stat -c '%a*%S/1024/1024' -f .`))
        echo free ${free}M
        overlays=""
        overlay_remove
        for d in $devices; do
                b=$(basename $d)
                size_bkl=$(blockdev --getsz $d) # in 512 blocks/sectors
                # reserve 1M space for snapshot header
                # ext3 max file length is 2TB   
                truncate -s$((((size_bkl+1)/2)+1024))K $b.ovr || (echo "Do you use ext4?"; return 1)
                loop=$(losetup -f --show -- $b.ovr)
                # https://www.kernel.org/doc/Documentation/device-mapper/snapshot.txt
                dmsetup create $b --table "0 $size_bkl snapshot $d $loop P 8"
                echo $d $((size_bkl/2048))M $loop /dev/mapper/$b
                overlays="$overlays /dev/mapper/$b"
        done
        overlays=${overlays# }
}

overlay_remove()
{
        for d in $devices; do
                b=$(basename $d)
                [ -e /dev/mapper/$b ] && dmsetup remove $b && echo /dev/mapper/$b 
                if [ -e $b.ovr ]; then
                        echo $b.ovr
                        l=$(losetup -j $b.ovr | cut -d : -f1)
                        echo $l
                        [ -n "$l" ] && losetup -d $(losetup -j $b.ovr | cut -d : -f1)
                        rm -f $b.ovr &> /dev/null
                fi
        done
}

Optional: figure out what happened

The Update time tells us which drive failed when:
 $ parallel --tag -k mdadm -E ::: $OVERLAYS|grep -E 'Update'
 /dev/mapper/sdq1            Update Time : Sat May  4 15:32:43 2013 # 3rd to fail
 /dev/mapper/sds1            Update Time : Sat May  4 15:32:03 2013 # 2nd to fail
 /dev/mapper/sdt1            Update Time : Sat May  4 15:29:47 2013 # 1st to fail
 /dev/mapper/sdu1            Update Time : Sat May  4 15:32:49 2013
 /dev/mapper/sdv1            Update Time : Sat May  4 15:32:49 2013
 /dev/mapper/sdw1            Update Time : Sat May  4 15:32:49 2013
Looking at each harddisk's Role it is clear that the 3 devices that failed were indeed data devices. The spare did not fail:
 $ parallel --tag -k mdadm -E ::: $OVERLAYS|grep -E 'Role'
 /dev/mapper/sdq1           Device Role : Active device 1 # 3rd to fail
 /dev/mapper/sds1           Device Role : Active device 0 # 2nd to fail
 /dev/mapper/sdt1           Device Role : Active device 3 # 1st to fail
 /dev/mapper/sdu1           Device Role : Active device 2
 /dev/mapper/sdv1           Device Role : spare
 /dev/mapper/sdw1           Device Role : Active device 4
So we are interested in assembling a RAID with the devices that were active last (sdu1, sdw1) and the last to fail (sdq1).

Force assembly

By forcing the assembly you can make mdadm clear the faulty state:
 $ mdadm --assemble --force /dev/md1 $OVERLAYS
 mdadm: forcing event count in /dev/mapper/sdq1(1) from 143 upto 148
 mdadm: clearing FAULTY flag for device 4 in /dev/md1 for /dev/mapper/sdv1
 mdadm: Marking array /dev/md1 as 'clean'
 mdadm: /dev/md1 has been started with 3 drives (out of 5) and 1 spare.
Rebuild will now start:
 $ cat /proc/mdstat 
 Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
 md1 : active raid6 dm-0[1] dm-4[6] dm-5[5] dm-3[2]
     305664 blocks super 1.2 level 6, 512k chunk, algorithm 2 [5/3] [_UU_U]
     [==>..................]  recovery = 11.5% (12284/101888) finish=0.4min speed=3071K/sec
It will rebuild on the overlay file, so you should pause the rebuild as the overlay file will otherwise eat your disk space:
 echo 0 > /proc/sys/dev/raid/speed_limit_max
 echo 0 > /proc/sys/dev/raid/speed_limit_min
You can add back the remaining drives as spares:
 $ parallel -j1 mdadm --add /dev/md1 ::: $OVERLAYS
 mdadm: Cannot open /dev/mapper/sdv1: Device or resource busy
 $ cat /proc/mdstat 
 Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
 md1 : active raid6 dm-2[8](S) dm-1[7] dm-0[1] dm-4[6] dm-5[5] dm-3[2]
     305664 blocks super 1.2 level 6, 512k chunk, algorithm 2 [5/5] [UUUUU]

Reset assembly

You may need to roll back the assembly. Do that by:
 mdadm --stop /dev/md1

File system check

You now have an assembled RAID. But we now need to figure out if the filesystem is still OK.

XFS

XFS stores a log that it replays on mount. This should be done before trying to repair the file system:
 mount /dev/md1 /mnt/md1
 # DO NOT USE THE FILESYSTEM, BUT IMMEDIATELY UMOUNT
 umount /mnt/md1
In certain situations the filesystem will crash your computer if used before it has been repaired.
 xfs_repair /dev/md1
If xfs_repair fails, try with -L:
 xfs_repair -L /dev/md1

Other file systems

Run fsck on the RAID-device:
 fsck /dev/md1
If there are load of errors:
 fsck -y /dev/md1

Examine the filesystem

After fixing the filesystem it is time to see if data survived. Mount the file system:
 mount /dev/md1 /mnt/md1
And examine /mnt/md1. Do not write to it, as everything you write will go into the overlay files.
If there are problems: reset the assembly, reset the overlay files and try different options. As long as you use the overlay files, it will be hard to destroy anything.
If everything is fine, you can now optionally make a backup before resetting the assembly, resetting the overlay files and do the fixing procedure on $DEVICES instead of $OVERLAYS. Congratulations: You just saved your data from a RAID failure.
Posted by at No comments:

Monday, January 4, 2016

Team Member traits

ive key "dynamics" to a successful team:
The findings are bolstered by academic research out of the University of Notre Dame's business school that was published earlier this fall in the Academy of Management Journal. The study looked at different teams at six companies, and found that work groups do better when members are motivated to help each other. In other words, self-interest will only take you so far at work.
When Google was just a little startup that helped you search the Internet for stuff, the company was pretty strict about hiring: Ivy League grads with high SAT scores were the preferred worker bees.
GOOGLE
The company figured out that was ridiculous quite quickly, as its people chief, Laszlo Bock, explained in his recent book Work Rules.
"Not shocking to you, perhaps, but these were early days at Google and, quite frankly, our approach was more elitist then," he writes. Now the company looks for bright, hardworking candidates who have demonstrated "resilience and an ability to overcome hardship." 
This latest bit of research feels like a natural step toward moving away from this automatic elitism -- the kind that reinforces a lack of diversity, by the way. Google is releasing the information publicly through its re:Work website, which is dedicated to sharing what it has learned about how to treat its nearly 60,000 employees. 
"We were pretty confident that we'd find the perfect mix of individual traits and skills necessary for a stellar team -- take one Rhodes Scholar, two extroverts, one engineer who rocks at AngularJS, and a PhD. Voila. Dream team assembled, right?" Juliai Rozovsky, an analyst in the Google's people operations (i.e., HR) department, writes in a blog post. "We were dead wrong. Who is on a team matters less than how the team members interact, structure their work, and view their contributions."
Veja a seguir as características mencionadas por Cook e comentadas por Carmine Gallo, autor do livro “The Apple Experience”, em artigo para a Forbes:
1. Idealismo
Visionário e apaixonado, Steve Jobs criou uma cultura de comprometimento emocional com o trabalho. Não à toa, paixão é palavra fácil na boca dos recrutadores da Apple. Ter uma “personalidade magnética” é tão valorizado por eles quanto esbanjar conhecimento técnico, comenta Gallo.
2. Obstinação
Cook diz que a Apple procura pessoas que não aceitam “não” como resposta. Na prática, isso significa ter opiniões fortes, debater ideias sem medo e dar feedbacks corajosos quando há necessidade de corrigir algo. “Todo dia eu estou cercado de pessoas que não concordam comigo”, comenta o CEO da empresa.
3. Pensamento original
Em uma famosa campanha publicitária da Apple nos anos 90, Steve Jobs sugeria que os clientes da marca “pensavam diferente”. O vídeo se referia a grandes personalidades do século 20, como Albert Einstein e Bob Dylan, como “pessoas loucas o suficiente para achar que poderiam mudar o mundo”. A comparação também vale para os candidatos ideais a uma vaga na empresa. “Queremos pessoas que não aceitam o status quo”, diz Tim Cook.
4. Insatisfação
Se você acha que tudo está muito bem, obrigado, talvez suas chances com um recrutador da Apple não sejam tão promissoras. Cook diz que o funcionário ideal é aquele que está descontente com a realidade, e está determinado a aperfeiçoá-la. Mas atenção: insatisfação crônica é diferente de prepotência. De acordo com Gallo, a empresa não quer um profissional que acredita ter todas as respostas, mas sim aquele que está disposto a descobri-las.
5. Descrença no impossível
Mais uma vez, a herança de Steve Jobs se faz presente. Em 2001, explica Gallo, quase todo consultor de varejo dizia ao fundador da Apple que lojas da marca não funcionariam. O pessimismo foi desmentido pela realidade - na verdade, pela insistência de Jobs. Hoje, quem mira uma vaga na empresa ganha pontos se tiver uma história que reflita essa característica: um desejo de provar que o "impossível" é perfeitamente realizável.

Posted by at No comments:

Tuesday, December 8, 2015

Mastering Strategy

The careers of superstar CEOs Bill Gates, Andy Grove, and Steve Jobs offer important lessons about how to become a better strategist.
Many discussions of strategy revolve around companies. But what about the people who develop corporate strategies? How can executives develop their skills as strategists?
There’s no better way than to learn from the masters. That’s the idea behind a recent book by David B. Yoffie, the Max and Doris Starr Professor of International Business Administration at Harvard Business School, and Michael A. Cusumano, the Sloan Management Review Distinguished Professor of Management at the MIT Sloan School of Management. Both men are experts in business strategy — they’ve been teaching the subject for nearly 30 years at Harvard and MIT, respectively. What’s more, Yoffie and Cusumano have studied or worked closely with some of the world’s leading technology executives. In their book, Strategy Rules: Five Timeless Lessons From Bill Gates, Andy Grove, and Steve Jobs (HarperCollins, 2015), Yoffie and Cusumano explore strategy insights drawn from the careers of the former CEOs of Microsoft Corp., Intel Corp., and Apple Inc.
MIT Sloan Management Review editorial director Martha E. Mangelsdorf spoke with Yoffie and Cusumano about what executives can learn from Gates, Grove, and Jobs about mastering the art of strategy. What follows is an edited and condensed version of that conversation.
MIT Sloan Management Review: Your book “Strategy Rules” looks at strategy lessons from three iconic CEOs from the computer industry: Bill Gates, Andy Grove, and Steve Jobs. What made you choose those three CEOs to write about?
Yoffie: There are several reasons for choosing these three. First and foremost, all three of the companies they led — Microsoft, Intel, and Apple — became the most valuable company in the world at some point. We were looking at three companies where we believed it was undisputed that the three individual CEOs had accomplished an extraordinary amount over their careers.
Second, we had spent a lot of time working with or observing all three of these individuals. We knew them, their records, and their companies extremely well. And by identifying three executives whose legacy as CEO was complete — and where the companies they led had continued to perform well after their departure — we could capture a more complete picture. We also understood the problems and occasional failures of these three CEOs, not just their successes.
That was an interesting aspect of the book. It was fun to learn about not just the things that we know about these three CEOs’ successes, but also about some of the things that didn’t go as well and the areas where they changed and developed as executives.
One of the ideas that I found intriguing in your book was your observation that strategic thinking is a capability that leaders develop over time — and that these executives, whom we know as having made some great decisions, didn’t necessarily start off as such accomplished strategists. Say a little bit about that.
Yoffie: Andy Grove is probably the easiest one to talk about, because he started as a scientist in the lab doing R&D. When he took the next step in his career to help launch and build Intel, he was a true operating manager. He was running a division and then became chief operating officer. Grove’s 1983 book, High Output Management, was all about making middle managers more effective; Grove didn’t become an effective strategist until several years into being CEO. Part of what enabled Grove’s development and transformation was his desire to learn new things — to continually go beyond his current capabilities.
For example, Andy was an engineer who initially knew less than nothing about brands. He was selling an industrial product — semiconductors for computers and other electronic devices — to other industrial companies, and the concept of building a consumer brand was far beyond his experience. But he made a large personal effort to educate himself to understand what a consumer brand was and how consumer pull could work for an industrial products company like Intel.
That whole learning process helped lead to the “Intel Inside” marketing campaign in 1991, which made Intel into one of the most valuable brands in the world.
Cusumano: To continue on this question of developing as a strategist, we believe that Bill Gates was a natural strategist and was born to be a strategic thinker. But again, he, too, learned: He learned to expand his horizons. The famous anecdote about Gates is that when IBM first came to him in 1980 for an operating system for IBM’s new personal computer, he sent them off to another company run by Gary Kildall. But when IBM came back to Gates, he clearly understood the opportunity that was ahead — to create the foundation for a whole new industry. Over the years, we’ve seen many examples of brilliant strategic moves by Gates: breaking with IBM; putting his resources behind Windows; embracing and extending the Internet; and then rebuilding Windows and Microsoft Office around the Internet.
What Gates really learned about is execution and organization. He learned that he couldn’t personally run whole areas of the company. He understood coding and algorithms, which allowed him to go one-on-one with engineers, but he went outside the company to hire talented managers with different backgrounds and experiences to run operations and various product groups.
Steve Jobs, on the other hand, always had great product instincts, but he had to learn to master strategy in the high-tech world. The strategy of Apple initially was great products, one product at a time. He only gradually adapted, with pressure from his management team in the 2000s. After resisting for years, Jobs eventually agreed to adopt a broader platform strategy, with a vision of a digital hub that targeted Windows as well as Macintosh users. By the time iPods and iTunes starting rolling out to the broader market beyond Macintosh users, we think Jobs had become a brilliant strategist.
Yoffie: I would also add that Steve was not pragmatic in his first 10 years as Apple’s CEO and that part of what made him a much better strategist by the late 1990s was he became a pragmatist. He recognized that you have to make deals with the enemy — in Apple’s case, Microsoft — and you have to delegate.
In your book, you identify five important strategy lessons that you drew from these three executives’ careers. Can you explain a little bit about the five strategy rules you identified? Let’s go through all five.

Strategy Rule #1: Look Forward, Reason Back

Yoffie: For managers, it’s a natural instinct to look backward and then reason forward about what they need to do today. That involves learning from history, thinking about the problems the business had yesterday, and how to solve similar problems tomorrow. But great strategists are like great chess players or great game theorists: They need to think several steps ahead towards the end of the game and then reason back to what that means about what they need to do today.
As a strategist, you need to think about where you want your business to be two, three, five, seven years down the road and then figure out what are the priorities and boundaries of what you need to do as a company today to get there. You need to be able to anticipate customer needs — not just solving the customer problems of today, but what the customer is going to need tomorrow. Then match that to the capabilities you can deliver in terms of new products and new processes for the customer over the next several years. You also need to anticipate what competitors will do and try to find ways to systematically build barriers to imitation and barriers to entry to reduce the likelihood that competitors will take away your advantage down the road. Finally, you have to be able to think about how whole industries may change.
The core story is a discipline of thinking several steps ahead and then figuring out what that means for the company now. This was a discipline that we saw across all three CEOs.
Cusumano: For example, all three of the CEOs extrapolated from Moore’s Law [the number of transistors on an integrated circuit doubling approximately every 18-24 months] in a different way. In the 1970s, both Gates and Jobs extrapolated that computing power was becoming ubiquitous and cheap, which would give birth to personal computers.
Gates, with Paul Allen in 1975, saw Moore’s Law and reasoned: Computers are just boxes without software. Software could be the source of value. Hardware is going to become a commodity. We’re going to control the software. And Microsoft was essentially the first software product company.
Jobs saw the same thing and concluded: Computers are going to be everywhere. We’re going to make them as easy to use as a typewriter or a toaster, out of the box. We’re going to make the computer a consumer appliance.
It took Grove a little while longer, but after a few years he figured out that Moore’s Law would lead to massive economies of scale and specialization in the computing industry, and that would probably make it difficult for the vertically integrated companies like IBM and Digital Equipment Corp. to maintain excellence and superiority in all the different segments. Grove reasoned that the computing industry would de-integrate into horizontal layers: Microsoft and some other companies would probably dominate software, but Intel would focus on the microprocessor. Eventually, he exited most other businesses, such as commodity memory products, and decided Intel was not going to build full computers and compete with its partners. Intel was just going to focus on that one layer: microprocessors.

Strategy Rule #2: Make Big Bets, Without Betting the Company

Cusumano: All three executives made big bets, but they never really bet the company. They always hedged those big bets. For example, one of Microsoft’s big bets was the decision to break with IBM in 1991. By the time Gates made that decision, Microsoft had many other companies as customers for DOS and then Windows — the PC clone industry. In addition, Microsoft had a small applications business that was growing quite fast. They also had the application business for the Macintosh. Breaking with IBM was a huge gamble, because Big Blue had really made Microsoft into a powerhouse, but Microsoft would not be killed by the divorce.

Strategy Rule #3: Build Platforms and Ecosystems — Not Just Products

Cusumano: These three executives set the intellectual foundations for understanding platform strategy and how it differs from product strategy. I can’t think of any three CEOs who have clarified our thinking more on that enormously important strategic concept.
For example, Gates understood the importance of platforms pretty much immediately in 1980, with Microsoft’s contract with IBM for the DOS operating system. Gates understood platforms through the lens of IBM; he knew IBM’s history quite well. The IBM mainframe had become an industry platform, where other companies had built compatible hardware. There were actual clones of the IBM mainframe. There were many peripherals that were clones, and there was a lot of software that was written to work on IBM and IBM-compatible machines.
Gates states very clearly — and we have the quote in the book — that he knew from day one, when Microsoft structured its deal with IBM to allow Microsoft to license DOS to other companies, that there would probably be a clone industry for personal computers, just like there had been for the IBM mainframe. So he saw that the personal computer would be a platform — and that Microsoft’s operating system could be a key element of that platform.

Strategy Rule #4: Exploit Leverage and Power — Play Judo and Sumo

Yoffie: If you’re going to be a great strategist, you’ve got to be able to execute at the tactical level. The things that you do every day, day-to-day with your customers, with your competitors, and with your partners become critical in your ability to execute your longer-term strategy. You have to be both clever and tough at the same time. The cleverness is the judo idea — trying to find ways to take advantage of your competitors’ strengths and turn them to your advantage, to find ways to avoid head-to-head struggles with your competitors at times when you’re not necessarily strong enough to compete that way.
Here’s an example of a judo tactic in strategy: When Jobs was launching iTunes, Apple only had around 2% of the PC market. So he used that to his advantage in negotiating with music executives; he persuaded them to license music on Apple’s terms as an experiment. In effect, he said: I’m only 2% of the market; what have you got to lose? In that case, being underestimated helped Steve Jobs get his way. That was, in retrospect, a huge mistake by the music industry.
Conversely, when you’re big enough and powerful enough, you have to be tough enough to extract as much value as possible in your interactions with other businesses. That’s the sumo aspect of strategy — not being afraid to throw your weight around. Gates, for example, regularly played hardball with customers and competitors alike. When he wanted Apple to adopt Internet Explorer in his efforts to win the browser wars, he threatened Gil Amelio, Apple’s then-CEO, with shutting down Microsoft Office for the Macintosh. Insiders believed this move could have put Apple out of business. Jobs was no less ruthless; for instance, he bullied book publishers into accepting Apple’s terms to launch the Apple iBookstore with the iPad. But as these two examples suggest, playing sumo means walking a fine line with antitrust. Both Microsoft and Apple faced antitrust battles with the U.S. Department of Justice.

Strategy Rule #5: Shape the Organization Around Your Personal Anchor

Cusumano: This was the hardest rule to describe. We try to explain why these three CEOs were effective at execution. Our answer was that each had a stake in the ground — a personal anchor — that helped them grow their companies, focus strategy, and hire people around.
For Gates, it was his understanding of software, which was as good as anybody in the world at the time he was launching Microsoft, at least for personal computers. For Jobs, it was his uncanny ability to understand the average user, especially the user interface. And Andy Grove had this incredible engineering-like process discipline; it was very clear that what he tried to do was bring the discipline of engineering to the messy business that semiconductor manufacturing was at that time. It was a business that was more art, or trial and error, than science and in which Intel was getting beaten up by the Japanese, who at that time were better at process. But Andy brought this incredibly disciplined and data-driven decision making, and intensity of debate and process discipline, to everything he did — to manufacturing, marketing, and sales, as well as to strategic planning.
Yoffie: Another aspect of a personal anchor is a bit paradoxical: You want to dive deep into the things you’re really good at, but at the same time stay at a high level and always keep the big picture in mind. You have to know yourself, know what you are good at, and know your weak spots. It doesn’t matter whether you’re an entrepreneur or running a $50 billion company; the key thing is figuring out how to compensate for your weaknesses in order to make the organization execute effectively. We think that’s true regardless of company size; any CEO has to do that. In the case of Grove and Gates, they knew very early on in their careers what they were good at and what they weren’t; their crisp execution depended on finding ways to get the right people around them to compensate for areas that weren’t their personal strengths.
In the case of Jobs, of course, he did not appreciate this ‘rule’ in the early days. One of the things he figured out when he came back to Apple the second time, in 1997, was that he really wasn’t good at many things, and he needed a lot of people to help him. He needed to figure out how to do what he did really well and drive that aspect of the organization — and then make sure he had other talented executives to lead areas such as supply chain and finance.
All three executives developed their skills as strategists over the years and became very strong in different ways by the end of their careers.
But as you point out in your book, none of these CEOs’ visions will last forever — particularly not in the fast-changing technology sector. How do these lessons apply to the next generation of technology CEOs? What’s different for the up-and-coming generation of technology entrepreneurs?
Cusumano: We think all of these principles and the details behind them — not just the high-level rules — are of extraordinary use to the next generation. I was recently at a lecture where I saw a list of the most valuable pre-IPO companies, and they’re nearly all platform companies. They’re taking advantage of exponential growth possible over the Internet in some shape or form, whether it’s Airbnb or Uber or many others.
And all of those businesses evolve from looking forward, reasoning back, figuring out what to do; making some big bets; building platforms rather than standalone products or services; figuring out how to be both clever and powerful; and building companies around the strengths of the founders. Most of these companies have a very distinct “edge” — a strategic and technical focus — to them that comes from the founders or the founders’ teams. But if entrepreneurs don’t then build broader teams with more diverse skills, they limit their success and growth.
I think all five principles are valuable for managers. But we also identified two additional lessons that apply to the next generation of entrepreneurs: First, while it is critical to build a company around your personal anchor and make that into an organizational anchor to sustain your advantage, you also have to beware that an anchor can limit you and hold the company back. And second, be aware of an inherent challenge associated with building a successful platform: You become so successful that you see the world through the lens of your platform. Then it gets very difficult to move to whatever comes next.
Posted by at No comments:

A Non-Geek’s Big Data Playbook

Introduction ...................................................................................... 4 terms & diagrams used in this playbook................................. 5 The Enterprise Data Warehouse.............................................................. 5 Big Data and Hadoop ............................................................................. 6
 play 1: stage structured data..................................................... 8
 play 2: process structured data ............................................. 10
 play 3: process non-integrated & unstructured data.... 11
 play 4: archive all data................................................................. 13
play 5: access all data via the EDW.......................................... 14
 play 6: access all data via Hadoop........................................... 16 conclusion........................................................................................ 18


Posted by at No comments:

Wednesday, October 7, 2015

6 tips for managing highly intelligent employees


Rubiks
IMAGE: FLICKR, THEILR
Quora-2129
BY QUORA
Don't manage: Guide. IQ and experience are two attributes you want your team members to over-index on, but there are plenty of other attributes that also matter, including judgment, work ethic, communication skills and teamwork. Unlike raw IQ, these are things that can be developed in any employee. Your job is to provide individualized guidance to each of your team members in the areas that they need to develop.
Here are some other general tips for guiding strong engineers.
  • Guide what they build, not how they build. As a manager, your job is to make sure your engineering team is working on the most important things. Your idea of what is most important will often differ from theirs, so you need to spend some time explaining why you feel your priorities are the right ones and letting them make the case for theirs. Once you've agreed on the priorities and outcomes you need, give them latitude to decide how they will achieve them.
  • Ask probing questions. Ask them what their design optimizes for, what their availability target is, what the latency requirements are, why they chose a particular persistence layer, why they chose the specific language, etc. Ask them what the standard in your company is for X or Y, and why they chose to diverge from the standard. Doing so will not only educate you, but also give you an opportunity to spot potential inconsistencies in their design or fuzzy thinking in general.
  • Connect them with very senior engineers even smarter or more experienced than they are, and set up a process to leverage that experience. This supports your engineers' learning and development and also ensures that their designs get an adequate architectural review if you're not equipped to perform one yourself.
  • Mediate arguments and (too) long-running discussions. Strong engineers have confidence in their design and coding judgment, and are inclined to get into debates with other engineers about the right way to do things. This is productive, to a point, but eventually it starts becoming unproductive and can create tension.
    Your job as a manager is to find ways to short-circuit arguments and long-running debates,
    Your job as a manager is to find ways to short-circuit arguments and long-running debates, which may mean listening to diverging viewpoints and then making a decision. If you're not equipped to make the decision, find a path to do so, which might mean reviewing with a very senior engineer not on your team and letting him or her make the decision.
  • Continue to hire team members smarter than you. You may want to balance a team of mostly experienced engineers with some more junior engineers, but you always want to hire for high raw technical IQ. Hiring an engineer is a long-term decision. Don't relax your standards in order to fill an engineering role quickly.
  • Manage out poor performers. Strong engineers don't like to work with weak ones. If you don't manage out poor performers, your team will accumulate them and your strong performers will move on to other teams or companies.
Posted by at No comments:
Subscribe to: Posts (Atom)