Well, recently I moved my development environment from Lubuntu 15.10 to 16.04 LTS. After installing XSDK and Petalinux 15.2 (yeah... I am still at 15.02 because I don't want my client to update to a new XSDK in the middle of the project and cause a number of problems...)
$ petalinux-build
INFO: Checking component...
INFO: Generating make files and build linux
...
[INFO ] install linux/kernel
[ERROR] ERROR: Invalid ELF file '/home/om/labcyte.git/lc-ddring/peta/plnx-ddring/images/linux/vmlinux'
[ERROR] make[1]: *** [package-subsystem-FIT] Error 255
ERROR: Failed to build linux
Hmm... What a bummer!
After searching for a while, I found this topic in xilinx forum
Following the advice on the link, I executed
$ sudo apt-get install tofrodos iproute tftpd-hpa gawk gcc git-core make
net-tools libncurses5-dev zlib1g-dev libssl-dev flex bison lib32z1
lib32ncurses5 lib32bz2-1.0 lib32stdc++6 libselinux1
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package lib32bz2-1.0
E: Couldn't find any package by glob 'lib32bz2-1.0'
E: Couldn't find any package by regex 'lib32bz2-1.0'
Hm... Looks like lib32bz2-1.0 is not present. Can it be renamed to something else?
Further searching using google yielded this link
The fix was easy.
sudo dpkg --add-architecture i386
root@E6420:~# apt-get update
...
apt-get install libbz2-1.0:i386
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
libbz2-1.0:i386
...
$ sudo apt-get install tofrodos iproute tftpd-hpa gawk gcc git-core make net-tools libncurses5-dev zlib1g-dev libssl-dev flex bison lib32z1 lib32ncurses5 lib32stdc++6 libselinux1
Reading package lists... Done
Building dependency tree
Reading state information... Done
bison is already the newest version (2:3.0.4.dfsg-1).
flex is already the newest version (2.6.0-11).
gawk is already the newest version (1:4.1.3+dfsg-0.1).
gcc is already the newest version (4:5.3.1-1ubuntu1).
libncurses5-dev is already the newest version (6.0+20160213-1ubuntu1).
libselinux1 is already the newest version (2.4-3build2).
make is already the newest version (4.1-6).
...
$ petalinux-build
...
[INFO ] post-install linux/rootfs/uWeb
[INFO ] package zImage
[INFO ] zImage linux/kernel
[INFO ] install linux/kernel
[INFO ] Package HDF bitstream
$
Great! Petalinux started building.
Wednesday, June 15, 2016
Monday, October 5, 2015
FreeRTOS on Zynq : recipe for target '*.s' failed
If you are working on FreeRTOS on Zynq, chances are high that you have come across this error while compiling FreeRTOS on Eclipse.
Running Make libs in ps7_cortexa9_0/libsrc/freertos822_xilinx_v1_0/src
make -C ps7_cortexa9_0/libsrc/freertos822_xilinx_v1_0/src -s libs "SHELL=/bin/sh" "COMPILER=arm-xilinx-eabi-gcc" "ARCHIVER=arm-xilinx-eabi-ar" "COMPILER_FLAGS= -O2 -c" "EXTRA_COMPILER_FLAGS=-g"
/bin/sh: arm-none-eabi-gcc: command not found
<builtin>: recipe for target '*.s' failed
make[1]: *** [*.s] Error 127
Makefile:27: recipe for target 'ps7_cortexa9_0/libsrc/freertos822_xilinx_v1_0/src/make.libs' failed
Fixing it is rather easy.
open the file Xilinx_FreeRTOS_BSP/repo/bsp/freertos822_xilinx_v1_0/src/Makefile_ps7_cortexa9
line 101 reads,
KERNEL_AR_OBJS = *.c *.S *.s
change it to read
KERNEL_AR_OBJS = *.c *.S
Save and exit
Re-Generate BSP sources and compile.
What happened is that, since there are no .s files at the very first go, make complains about it and halts processing. From next time onwards, since there are .s files created in the previous compilations, it does not complain or stop processing.
Looks like this is a bug in FreeRTOS makefile (In microblaze make file, there is no .s in the corresponding line)
Running Make libs in ps7_cortexa9_0/libsrc/freertos822_xilinx_v1_0/src
make -C ps7_cortexa9_0/libsrc/freertos822_xilinx_v1_0/src -s libs "SHELL=/bin/sh" "COMPILER=arm-xilinx-eabi-gcc" "ARCHIVER=arm-xilinx-eabi-ar" "COMPILER_FLAGS= -O2 -c" "EXTRA_COMPILER_FLAGS=-g"
/bin/sh: arm-none-eabi-gcc: command not found
<builtin>: recipe for target '*.s' failed
make[1]: *** [*.s] Error 127
Makefile:27: recipe for target 'ps7_cortexa9_0/libsrc/freertos822_xilinx_v1_0/src/make.libs' failed
Fixing it is rather easy.
open the file Xilinx_FreeRTOS_BSP/repo/bsp/freertos822_xilinx_v1_0/src/Makefile_ps7_cortexa9
line 101 reads,
KERNEL_AR_OBJS = *.c *.S *.s
change it to read
KERNEL_AR_OBJS = *.c *.S
Save and exit
Re-Generate BSP sources and compile.
What happened is that, since there are no .s files at the very first go, make complains about it and halts processing. From next time onwards, since there are .s files created in the previous compilations, it does not complain or stop processing.
Looks like this is a bug in FreeRTOS makefile (In microblaze make file, there is no .s in the corresponding line)
Friday, September 25, 2015
Keeping your external hdd cooler (and last longer)
I have a 3TB external HDD -- it is a WDC WD30EZRX-00SPEB0 HDD in an USB3.0/eSATA 2.0 enclosure I bought from EBay. I have a cron job running to back up important git repos and other work I currently do
Since the cron job is scheduled to run every day 2 times, the drive sits idle for rest of the day. And after a couple of days, I noticed that the drive is getting really really warm
So, I started running hddtemp in my conky to keep a watch on the temperature and I realized that when I start my day,
$ hddtemp /dev/sdd
/dev/sdd: WDC WD30EZRX-00SPEB0: 30°C
After an hour or so,
$ hddtemp /dev/sdd
/dev/sdd: WDC WD30EZRX-00SPEB0: 48°C
Hmm... it is well within the operating range. (1-60, according to Spec here). But I don't like it. Must be using a lot of energy...
First I thought of keeping it switched off (the enclosure has a power switch). But I had to change my mind after 2 days... I keep the drive under the table and it was hard to reach. Also, I hit my head on the underside of the table ... ouch!
So, I want a mechanism to mount the drive when required and unmount it after I am done with my backup (BTW, I use luckybackp on Debian. It is a pretty neat tool for backing up data)
# apt-get install autofs
Open /etc/auto.master and insert the following line (or similar one)
/export/esata /etc/auto.scsi --timeout=60,user,defaults,exec,suid,dev
This line tells autofs to use the map /etc/auto.scsi with mount options user,defaults,exec,suid,dev and impose a time out of 60 seconds for the mount point /export/esata
$ cat /etc/auto.scsi
wdgreen -fstype=auto :/dev/esata5
This tells automount to mount /dev/esata5 onto the directory wdgreen in /export/esata/ with file system type set to auto (will resolve to ext4, since I use ext4 on the partition)
For another drive in another system, I have the following entry
$ cat /etc/auto.usb
seagate1-2 -fstype=auto :/dev/disk/by-id/ata-ST31000528AS_6VPFKQAN-part2
Hmm... but how to make sure that the drive, when connected, will create a dev entry /dev/esata5?
[[By using /dev/disk/by-id/* entries, you can avoid the following udevadm gymnastics.]]
Udev is the answer
# udevadm info --query=name --attribute-walk /dev/sdd |less # replace sdd with your drive's block device
From the long list, pick an attribute that identifies the drive unambiguously. In my case, it is
ATTRS{model}=="WDC WD30EZRX-00SPEB0"
under
SUBSYSTEMS=="scsi"
A word of caution : the spaces within double quotes are important, do not miss them if you have space(s) at the beginning or end
Use this string to create a udev rule
# cat /etc/udev/rules.d/89-esata-disk.rules
SUBSYSTEMS=="scsi", ATTRS{model}=="WDC WD30EZRX-00SPEB0", SYMLINK+="esata%n"
Save the rule and restart udev (Sometimes you may need to restart the system, I have no idea why)
Now all is set
$ ls /export/esata
<nothing>
$ cd /export/esata/wdgreen
<will take a few seconds, you may hear the drive spinning up as well>
$ ls
<file list here>
Similarly,
$ cd /export/usbdisk/seagate1-2
$ ls
<file list here>
The auto.usb for that drive is here
$ cat /etc/auto.usb
seagate1-2 -fstype=auto :/dev/disk/by-id/ata-ST31000528AS_6VPFKQAN-part2
# hdparm -y /dev/esata
/dev/esata:
issuing standby command
Shedule it in cron
# crontab -e
5,25,45 * * * * /sbin/hdparm -y /dev/
This means cron will issue the command to sleep every hour 5th, 25th and 45th minutes
You can run it at the interval you want. But please note that once the drive is in sleep mode, it takes 5-15 seconds to spin up and be accessible
After all these, when I check the temperature,
# hddtemp /dev/esata
/dev/esata: WDC WD30EZRX-00SPEB0 : drive is sleeping
# cd /export/esata/wdgreen
<takes around 10 seconds here ...>
$ hddtemp /dev/sdd
/dev/sdd: WDC WD30EZRX-00SPEB0: 34°C
Success!
Edit:
Apparently hdparm -y applies only to HGST drives (hitachi)
For other drives (e.g. ones connected over USB 3.0, you can use smartctl)
$ if [ -L /dev/disk/by-id/ata-ST31000528AS_6VPFKQAN ] ; then smartctl -d sat -s standby,60 /dev/disk/by-id/ata-ST31000528AS_6VPFKQAN ; fi
This commands sets the drive standby time to 300 seconds.
-d sat is for some USB to SATA chipsets. check smartctl for other possible values to use with -d option
Since the cron job is scheduled to run every day 2 times, the drive sits idle for rest of the day. And after a couple of days, I noticed that the drive is getting really really warm
So, I started running hddtemp in my conky to keep a watch on the temperature and I realized that when I start my day,
$ hddtemp /dev/sdd
/dev/sdd: WDC WD30EZRX-00SPEB0: 30°C
After an hour or so,
$ hddtemp /dev/sdd
/dev/sdd: WDC WD30EZRX-00SPEB0: 48°C
Hmm... it is well within the operating range. (1-60, according to Spec here). But I don't like it. Must be using a lot of energy...
First I thought of keeping it switched off (the enclosure has a power switch). But I had to change my mind after 2 days... I keep the drive under the table and it was hard to reach. Also, I hit my head on the underside of the table ... ouch!
So, I want a mechanism to mount the drive when required and unmount it after I am done with my backup (BTW, I use luckybackp on Debian. It is a pretty neat tool for backing up data)
configuring autofs on debian/ubuntu
Install autofs# apt-get install autofs
Open /etc/auto.master and insert the following line (or similar one)
/export/esata /etc/auto.scsi --timeout=60,user,defaults,exec,suid,dev
This line tells autofs to use the map /etc/auto.scsi with mount options user,defaults,exec,suid,dev and impose a time out of 60 seconds for the mount point /export/esata
$ cat /etc/auto.scsi
wdgreen -fstype=auto :/dev/esata5
This tells automount to mount /dev/esata5 onto the directory wdgreen in /export/esata/ with file system type set to auto (will resolve to ext4, since I use ext4 on the partition)
For another drive in another system, I have the following entry
$ cat /etc/auto.usb
seagate1-2 -fstype=auto :/dev/disk/by-id/ata-ST31000528AS_6VPFKQAN-part2
Hmm... but how to make sure that the drive, when connected, will create a dev entry /dev/esata5?
[[By using /dev/disk/by-id/* entries, you can avoid the following udevadm gymnastics.]]
Udev is the answer
# udevadm info --query=name --attribute-walk /dev/sdd |less # replace sdd with your drive's block device
From the long list, pick an attribute that identifies the drive unambiguously. In my case, it is
ATTRS{model}=="WDC WD30EZRX-00SPEB0"
under
SUBSYSTEMS=="scsi"
A word of caution : the spaces within double quotes are important, do not miss them if you have space(s) at the beginning or end
Use this string to create a udev rule
# cat /etc/udev/rules.d/89-esata-disk.rules
SUBSYSTEMS=="scsi", ATTRS{model}=="WDC WD30EZRX-00SPEB0", SYMLINK+="esata%n"
Save the rule and restart udev (Sometimes you may need to restart the system, I have no idea why)
Now all is set
$ ls /export/esata
<nothing>
$ cd /export/esata/wdgreen
<will take a few seconds, you may hear the drive spinning up as well>
$ ls
<file list here>
Similarly,
$ cd /export/usbdisk/seagate1-2
$ ls
<file list here>
The auto.usb for that drive is here
$ cat /etc/auto.usb
seagate1-2 -fstype=auto :/dev/disk/by-id/ata-ST31000528AS_6VPFKQAN-part2
Putting the drive to sleep
# hdparm -y /dev/esata
/dev/esata:
issuing standby command
Shedule it in cron
# crontab -e
5,25,45 * * * * /sbin/hdparm -y /dev/
This means cron will issue the command to sleep every hour 5th, 25th and 45th minutes
You can run it at the interval you want. But please note that once the drive is in sleep mode, it takes 5-15 seconds to spin up and be accessible
After all these, when I check the temperature,
# hddtemp /dev/esata
/dev/esata: WDC WD30EZRX-00SPEB0 : drive is sleeping
# cd /export/esata/wdgreen
<takes around 10 seconds here ...>
$ hddtemp /dev/sdd
/dev/sdd: WDC WD30EZRX-00SPEB0: 34°C
Success!
Edit:
Apparently hdparm -y applies only to HGST drives (hitachi)
For other drives (e.g. ones connected over USB 3.0, you can use smartctl)
$ if [ -L /dev/disk/by-id/ata-ST31000528AS_6VPFKQAN ] ; then smartctl -d sat -s standby,60 /dev/disk/by-id/ata-ST31000528AS_6VPFKQAN ; fi
This commands sets the drive standby time to 300 seconds.
-d sat is for some USB to SATA chipsets. check smartctl for other possible values to use with -d option
Sunday, September 20, 2015
awk: symbol lookup error: awk: undefined symbol: mpfr_z_sub
Well, I have been working with Vivado 2014.4 for a while for a client's microZed board. After upgrading to Vivado (and XSDK) 2015.2, I started seeing this error
[1253]$ cat !!:$
cat ~/bin/xilinx-2015_2-setup.sh
#! /bin/bash
if [ "X$(basename -- "$0")" = "Xsetup-xilinx-sdk.sh" ] ; then
echo "$0 must be sourced. Cannot execute"
exit -1
fi
# set -x
XILINX_SDK_PATH=/export/space/xilinx/opt/SDK/2015.2
XILINX_UBOOT_SRC=/export/space/xilinx/src/u-boot-xlnx.git
XILINX_LINUX_SRC=/export/space/xilinx/src/linux-xlnx.git
XILINX_PETA_PATH=/export/space/xilinx/petalinux-v2015.2-final
export CROSS_COMPILE=arm-xilinx-linux-gnueabi-
echo $PATH | grep -q ${XILINX_SDK_PATH}
if [ $? != 0 ] ; then
PATH=$PATH:${XILINX_SDK_PATH}/bin:/export/space/xilinx/petalinux-v2015.2-final/tools/linux-i386/petalinux/bin/unexport:/export/space/xilinx/petalinux-v2015.2-final/tools/linux-i386/petalinux/bin:${XILINX_UBOOT_SRC}/tools
echo "PATH=${PATH}"
fi
source ${XILINX_SDK_PATH}/settings64.sh
source ${XILINX_PETA_PATH}/settings.sh
#set +x
$source ~/bin/xilinx-setup
[1277]$ petalinux-config --get-hw-description=../hwdef
awk: symbol lookup error: awk: undefined symbol: mpfr_z_sub
awk: symbol lookup error: awk: undefined symbol: mpfr_z_sub
INFO: Checking component...
awk: symbol lookup error: awk: undefined symbol: mpfr_z_sub
...
Repeats a number of times.
Now, I am stuck because petalinux-project fails with these messages.
After some web searches, I came across the following site where this is documented - googoolia. The suggestion is to do
LD_LIBRARY_PATH=”” awk
I tried whether it would work with petalinux-project,
[1282]$ LD_LIBRARY_PATH="" petalinux-config --get-hw-description=../hwdef
INFO: Checking component...
INFO: Getting hardware description...
hooray! It did
Later I thought about a better method
$ ldd `which awk` | grep Vivado
libmpfr.so.4 => /export/space/xilinx/opt/Vivado/2015.2/lib/lnx64.o/libmpfr.so.4 (0x00007f8b1e72e000)
libgmp.so.7 => /export/space/xilinx/opt/Vivado/2015.2/lib/lnx64.o/libgmp.so.7 (0x00007f8b1d0c0000)
[1283]$ pushd /export/space/xilinx/opt/Vivado/2015.2/lib/lnx64.o
/export/space/xilinx/opt/Vivado/2015.2/lib/lnx64.o ~/labcyte.git/peta/plnx-project
[1284]$ mv libmpfr.so.4 libmpfr.so.4.orig
[1285]$ mv libgmp.so.7 libgmp.so.7.orig
[1286]$ ldd `which awk` | grep Vivado
[1289]$ petalinux-config --get-hw-description=../hwdef
INFO: Checking component...
INFO: Getting hardware description...
Good, I eliminated some keystrokes :-D
[1253]$ cat !!:$
cat ~/bin/xilinx-2015_2-setup.sh
#! /bin/bash
if [ "X$(basename -- "$0")" = "Xsetup-xilinx-sdk.sh" ] ; then
echo "$0 must be sourced. Cannot execute"
exit -1
fi
# set -x
XILINX_SDK_PATH=/export/space/xilinx/opt/SDK/2015.2
XILINX_UBOOT_SRC=/export/space/xilinx/src/u-boot-xlnx.git
XILINX_LINUX_SRC=/export/space/xilinx/src/linux-xlnx.git
XILINX_PETA_PATH=/export/space/xilinx/petalinux-v2015.2-final
export CROSS_COMPILE=arm-xilinx-linux-gnueabi-
echo $PATH | grep -q ${XILINX_SDK_PATH}
if [ $? != 0 ] ; then
PATH=$PATH:${XILINX_SDK_PATH}/bin:/export/space/xilinx/petalinux-v2015.2-final/tools/linux-i386/petalinux/bin/unexport:/export/space/xilinx/petalinux-v2015.2-final/tools/linux-i386/petalinux/bin:${XILINX_UBOOT_SRC}/tools
echo "PATH=${PATH}"
fi
source ${XILINX_SDK_PATH}/settings64.sh
source ${XILINX_PETA_PATH}/settings.sh
#set +x
$source ~/bin/xilinx-setup
[1277]$ petalinux-config --get-hw-description=../hwdef
awk: symbol lookup error: awk: undefined symbol: mpfr_z_sub
awk: symbol lookup error: awk: undefined symbol: mpfr_z_sub
INFO: Checking component...
awk: symbol lookup error: awk: undefined symbol: mpfr_z_sub
...
Repeats a number of times.
Now, I am stuck because petalinux-project fails with these messages.
After some web searches, I came across the following site where this is documented - googoolia. The suggestion is to do
LD_LIBRARY_PATH=”” awk
I tried whether it would work with petalinux-project,
[1282]$ LD_LIBRARY_PATH="" petalinux-config --get-hw-description=../hwdef
INFO: Checking component...
INFO: Getting hardware description...
hooray! It did
Later I thought about a better method
$ ldd `which awk` | grep Vivado
libmpfr.so.4 => /export/space/xilinx/opt/Vivado/2015.2/lib/lnx64.o/libmpfr.so.4 (0x00007f8b1e72e000)
libgmp.so.7 => /export/space/xilinx/opt/Vivado/2015.2/lib/lnx64.o/libgmp.so.7 (0x00007f8b1d0c0000)
[1283]$ pushd /export/space/xilinx/opt/Vivado/2015.2/lib/lnx64.o
/export/space/xilinx/opt/Vivado/2015.2/lib/lnx64.o ~/labcyte.git/peta/plnx-project
[1284]$ mv libmpfr.so.4 libmpfr.so.4.orig
[1285]$ mv libgmp.so.7 libgmp.so.7.orig
[1286]$ ldd `which awk` | grep Vivado
[1289]$ petalinux-config --get-hw-description=../hwdef
INFO: Checking component...
INFO: Getting hardware description...
Good, I eliminated some keystrokes :-D
Sunday, August 10, 2014
Cross compiling mtd tools
I need to compile mtd-tools for arm, to be precise Xilinx ZynQ z702 which, by the way, a very nice platform for moderately complex projects. The presence of an integrated FPGA is the key aspect of its "all programmable" tag-line. You can find more info on ZynQ HERE
I am currently working on ZynQ derivative and
going to test the QSPI flash interface to NOR, and possibly the NAND
interface (if the board is wired up). In any case, I need mtd tools
because,
- There is no n/w i/f on the target board to download the linux kernel image
- Our linux kernel require to be run from flash (jffs2, jffs2 being the better choice?) root file system, not as a ramfs (ext2/3/4)
During the process, I was referring to MTD
compilation notes at
this TI page. But TI page is not very clear about cross
compilation. So, here are my thoughts and steps I took to get it
compiled.
Step 0: Get your cross compilation environment ready
Well, on ZynQ (or any other platform requiring
cross compilation), setting up the tool chain is a subject by
itself. Here I assume that you have (somehow) setup your cross
compilation environment and the tool chain is available to you as
${CROSS_COMPILE}<toolchain
binary>
Step1 : Get the source code
Mtd-utils require the following dependencies
satisfied for compilation
- zlib (compression)
- http://zlib.net/zlib-1.2.8.tar.gz – Latest as of the time of writing.
- lzo (compression) (Here is the wikipedia article on LZO)
- http://www.oberhumer.com/opensource/lzo/download/lzo-2.06.tar.gz – You may want to download a later version
- e2fsprogs (for uuid)
- http://git.kernel.org/cgit/fs/ext2/e2fsprogs.git – Click on the snapshot link to download. You may want to download a stable release
- mtd-utils
- http://git.infradead.org/mtd-utils.git – Click on the snapshot link to download. You may want to download a stable release
Once download is complete, gunzip/untar the files
into respective directories.
Prepare prefix directory
Zlib, lzo and e2fs progs being dependencies for
mtd-utils, you want to keep all these in a place accessible by
mtd-utils during compilation. That means, you need to install these
libraries and header files. By default, the prefix (/usr/local/...)
may not be a good idea because you are compiling these libraries for
a different architecture (arm in my case). I have chosen a target
prefix directory /export/prj/voler/mtd-utils/; you may choose
something similar.
$ echo $CROSS_COMPILE
/export/prj/Xilinx/SDK/2014.2/gnu/arm/lin/bin/arm-xilinx-linux-gnueabi-
[om@hexcore
/export/prj/voler/zlib-1.2.8]
$ arm-xilinx-linux-gnueabi-gcc
arm-xilinx-linux-gnueabi-gcc: fatal
error: no input files
compilation terminated.
Cross compile Zlib
[om@hexcore
/export/prj/voler/zlib-1.2.8]
$ ./configure
–prefix=/export/prj/voler/mtd-utils/
This configure does not support
cross compiling environment. So, we need to make manual changes to
generated Makefile like the following
CC={CROSS_COMPILE}gcc
....
LDSHARED={CROSS_COMPILE}gcc -shared
-Wl,-soname,libz.so.1,--version-script,zlib.map
CPP={CROSS_COMPILE}gcc -E
$make
....
[om@hexcore
/export/prj/voler/zlib-1.2.8]
$ file
/export/prj/voler/mtd-utils/lib/libz.so.1.2.8
/export/prj/voler/mtd-utils/lib/libz.so.1.2.8:
ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV),
dynamically linked, not stripped
Cross compile lzo
[om@hexcore
/export/prj/voler/lzo-2.06]
$ ./configure
--host=arm-xilinx-linux-gnueabi --prefix=/export/prj/voler/mtd-utils
configure: WARNING: if you wanted to
set the --build type, don't use --host.
If a cross compiler is detected
then cross compile mode will be used
configure: Configuring LZO 2.06
....
LZO configuration summary
-------------------------
LZO version :
2.06
configured for host :
arm-xilinx-linux-gnueabi
source code location : .
compiler :
arm-xilinx-linux-gnueabi-gcc
preprocessor definitions :
-DLZO_HAVE_CONFIG_H=1
preprocessor flags :
compiler flags : -g
-O2
build static library : yes
build shared library : no
enable i386 assembly code : no
LZO 2.06 configured.
[om@hexcore
/export/prj/voler/lzo-2.06]
$ make ; make install
make all-am
----------------------------------------------------------------------
Libraries have been installed in:
/export/prj/voler/mtd-utils/lib
Cross compile e2fsprogs
[om@hexcore
/export/prj/voler/e2fsprogs-1.42.11]
$ ./configure
--host=arm-xilinx-linux-gnueabi –prefix=/export/prj/voler/mtd-utils
[om@hexcore
/export/prj/voler/e2fsprogs-1.42.11]
$ make ; make install
cd lib/uuid/
make install
Cross compile mtd-utils
Mtd-utils has no ./configure script. That means,
we need to manually change the Makefile for cross compilation related
changes.
PREFIX
= /home/temp/mtdutils
CROSS=arm-linux-gnueabi-
ZLIBCPPFLAGS
= -I$(PREFIX)/include
ZLIBLDFLAGS
= -L$(PREFIX)/lib
LZOCPPFLAGS
= -I$(PREFIX)/include
LZOLDFLAGS
= -L$(PREFIX)/lib
LDFLAGS
+= $(ZLIBLDFLAGS) $(LZOLDFLAGS)
#
if you want the resulting binaries standalone binaries, add -static
here
#
LDFLAGS += $(ZLIBLDFLAGS) $(LZOLDFLAGS) -static
CFLAGS
?= -O2 -g $(ZLIBCPPFLAGS) $(LZOCPPFLAGS)
Edit the common.mk file and comment out the
PREFIX=/usr line. Then make and install:
[om@hexcore
/export/prj/voler/mtd-utils-9f10713]
WITHOUT_XATTR=1 make; make install
(Thanks to
http://wiki.beyondlogic.org/index.php/Cross_Compiling_MTD_Utils_for_ARM,
I did not have to re-invent this)
Precompiled binaries for ZynQ
Download precompiled binaries for ZynQ / ARM v7
from here :
|
Binary |
Size |
Md5sum |
sha256sum |
|---|---|---|---|
|
7676764
|
622e0d4bc7f3197c9dc0ccf00d93b498
|
0af126bc948522db9d821
1b9e479f41173c028e105
7bc27a02096957720f6bfb
|
|
|
11847584
|
f55e42e81a2e6944fa87aa0f97117ee0
|
f613c70337be8bbe316567
8715930e9cc004c18218d9
d9ed8f83ecf4acf04f56
|
A sample run here:
root@zynq:~# mtdinfo
Count of MTD devices: 5
Present MTD devices:
mtd0, mtd1, mtd2, mtd3, mtd4
Sysfs interface supported: yes
root@zynq:~# mtdinfo /dev/mtd0
mtd0
Name:
qspi-fsbl-uboot
Type: nor
Eraseblock size:
65536 bytes, 64.0 KiB
Amount of eraseblocks: 16
(1048576 bytes, 1024.0 KiB)
Minimum input/output unit size: 1
byte
Sub-page size: 1
byte
Character device major/minor:
90:0
Bad blocks are allowed:
false
Device is writable:
true
Wednesday, November 7, 2012
Power management primer : How to write a cpu frequency controlling driver?
You must be familiar with dynamically changing cpu frequencies. On x86 this has been a de-facto standard for a long time. But many of the embedded systems, where the importance of dynamic frequency scaling cannot be underestimated in terms of power consumption alone (just think of higher battery life or a million home routers working at half the power) are not actively doing this. Some lack hardware features to implement this feature, others do not simply have a driver that can use the hardware capabilities provided.
In this post, I would explain the innards of a cpu frequency scaling driver and its interface to Linux CPU frequency scaling framework.
Pre-requisites
- Obviously, first thing you might need to have a cpu that is capable of scaling the frequency up and down.
- The programmer's reference manual : Mostly CPU frequency and stuff like that would be buried deep inside the manual.
- There would be a base frequency (Mostly the CPU PLL frequency)
- There could be a multiplier and configuration registers to calculate the base frequency the cpu can run at. Similarly you can calcuate the highest frequency supported as well.
Complications do not end here, as there could be some relation between DRAM frequency and CPU frequency PLL. In that case you will have to figure out the supported combinations of CPU and DRAM frequencies. You might need help from board engineers to create the table. etc. etc. But, eventually, you need a supported frequency table and corresponding values to write in the configuration registers. With multi core cpus and SoCs, changing frequency of a cpu/core might have implications on the peripherals/other cores as well. You are warned :-)
Example
On one of the cpus I was working with, I had the following details- Clock divider value per core : each core controls four h/w threads
- PLL frequency
- Divider increment registers and Divider decrement registers
There were many constraints like some of the bits were read only, Increment reg. and decrement reg. were to be written with all 1s ..etc. You might face similar constraints as well. Assuming you have something similar to below table, we can proceed as follows.
| DIV value | Frequency(MHz) |
|---|---|
| 1 | 100 |
| 2 | 133 |
| 3 | 160 |
| 4 | 200 |
| 5 | 229 |
| 6 | 266 |
| 7 | 320 |
| 9 | 400 |
| 11 | 533 |
| 13 | 800 |
| 15 | 1200 |
| 17 | 1600 |
You may opt to support only a subset of these possible frequency ranges by dropping some DIV values. But you will not be able to support some frequency other than what your hardware actually supports.
Thus, the first step is to define an array of supported DIV values like below
static int xxx_divs[XXX_DIVS] = {1, 2, 3, 4, 5, 6, 7, 9, 11, 13, 15}; /* drop 17 */
static struct cpufreq_frequency_table xxx_freq_table[XXX_DIVS + 1];
Now, you may define your function that actually would manipulate the frequency
multiplers in the hw as below
u32 change_cpu_freq(int cpu, int dec)
{
/* body of the function that actually does the work */
return 0;
}
Next important step is to define a cpufreq_driver structure such that kernel can be made aware of the possible frequency support (i.e, what function to call to get the speed, set the speed ..etc). That is done by instantiating a cpufreq_driver structure like below
static struct cpufreq_driver xxx_cpufreq_driver = {
.owner = THIS_MODULE,
.flags = 0,
.verify = xxx_cpufreq_verify_speed,
.target = xxx_cpufreq_set_target,
.get = xxx_cpufreq_get_speed,
.init = xxx_cpufreq_driver_init,
.exit = xxx_cpufreq_driver_exit,
.name = "xxx-cpufreq",
};
There are some more fields to define if you prefer. Please take a look at
include/linux/cpufreq.h for more info. Plesae note that I have defined `target`
and not `setpolicy'
The init function for this module is,
static int __init xxx_cpufreq_init(void)
{
setup_affected_cpus(cpu_present_map);
if (build_cpufreq_table(xxx_freq_table) < 0) {
return -EFAULT;
}
return cpufreq_register_driver(&xxx_cpufreq_driver);
}
A few points to note here:
- I call a static function setup_affected_cpus() with the map of cpus present at the moment. I can do this because my h/w does not support cpu hotplugging. The objective of this function is to create a static map of cpu threads on the same core -- all of which will change frequency if any one of them changes the core frequency (all cpus in this set is driven by a single core). You may do it some different way.
- After registration, get/target/verify could be called anytime (after driver init => .init). So, I prepare the supported cpu frequency table in advance; well before calling the registration function. I guess you could do it in driver init as well
- The driver init function gets the policy structure where we can fill in details like related cpus. This function will be called for every cpu/core of the system. This is where you can setup values like transition_latency, affected cpus ..etc.
- There are a number of helper functions available @ drivers/cpufreq/freq_table.c for massaging the values you filled in cpu frequency table. You may opt to use validation functions like cpufreq_frequency_table_cpuinfo(), cpufreq_frequency_table_verify() ..etc
The workhorse function for cpu freq. driver is .target function (xxx_cpufreq_set_target() for me). This function will be passed policy, target cpu and another parameter 'relation' (which signifies whether to ceil() or floor() the value for optimal frequency). Depending on the current frequency and target frequency, you might have to call functions to incr/decr cpu frequency from this function. Most important to note is that you should call cpufreq_notify_transition(&freq, CPUFREQ_PRECHANGE) before you change the frequency and cpufreq_notify_transition(&freq, CPUFREQ_POSTCHANGE) after you have reached the target frequency.
This completes kernel driver part. You will have to enable cpu frequency governors if you wish to use them. Take a look at some examples using "userspace" governor as default
/sys/devices/system/cpu/cpu0/cpufreq # ls
affected_cpus scaling_cur_freq
cpuinfo_cur_freq scaling_driver
cpuinfo_max_freq scaling_governor
cpuinfo_min_freq scaling_max_freq
cpuinfo_transition_latency scaling_min_freq
related_cpus scaling_setspeed
scaling_available_governors
/sys/devices/system/cpu/cpu0/cpufreq # for I in *
> do
> echo $I
> cat $I
> echo "...Done"
> done
affected_cpus
0 1 2 3
...Done
cpuinfo_cur_freq
800000
...Done
cpuinfo_max_freq
800000
...Done
cpuinfo_min_freq
100000
...Done
cpuinfo_transition_latency
2000000
...Done
related_cpus
0 1 2 3
...Done
scaling_available_governors
conservative ondemand powersave userspace performance
...Done
scaling_cur_freq
800000
...Done
scaling_driver
xxx-cpufreq
...Done
scaling_governor
userspace
...Done
scaling_max_freq
800000
...Done
scaling_min_freq
100000
...Done
scaling_setspeed
800000
...Done
# Using unsupported (lower than lowest) value as frequency
/sys/devices/system/cpu/cpu0/cpufreq # echo 90000 > scaling_setspeed
/sys/devices/system/cpu/cpu0/cpufreq # cat cpuinfo_cur_freq
100000
#cpu set to correct lowest frequency
#CPU set to unsupported (intermediate) value as frequency
/sys/devices/system/cpu/cpu0/cpufreq # echo 120000 > scaling_setspeed
/sys/devices/system/cpu/cpu0/cpufreq # cat cpuinfo_cur_freq
133328
# Using unsupported (higher than highest) value as frequency
# Please note th capped highest 800M, even though h/w supports 1.2G
/sys/devices/system/cpu/cpu0/cpufreq # echo 1123000 > scaling_setspeed
/sys/devices/system/cpu/cpu0/cpufreq # cat cpuinfo_cur_freq
800000
#CPU set to supported value as frequency
/sys/devices/system/cpu/cpu0/cpufreq # echo 400000 > scaling_setspeed
/sys/devices/system/cpu/cpu0/cpufreq # cat cpuinfo_cur_freq
400000
There are finer points to consider : like if your system clock is dependent on
cpu frequency, you might need to change the clocksource to a constant frequency
timer. That will be explained in a future post. Stay tuned...
Wednesday, October 31, 2012
how to fix that PCI(e) memspace?
That was a curious problem : a customer wanted hotplug functionality work with
our MIPS based board. Although we support native hotplug (by virtue of the
controller supporting native hotplug), the board did not have any hot-plug
support infrastructure : no attention button, no power control to slots ..etc.
So, we had to resort to a PLX chipset -- PEX 8624 and RDK board.
Thus, I started with an intel eepro (PCI ID - 8086:10d3) pcie single lane 1G ethernet
card. The driver has been available with linux and it is rock stable. After
compiling it as a module, (e1000e.ko) Linux was booted on the board.
Here is the topology
MIPS -> PCIe controller -> PEX 8624 -> eepro card
There are four 8x pcie controllers (yeah.. it can do 160Gbps sustained). Our
boot loader initializes the controllers and sets up BARs..etc. Linux is told not
to reconfigure (re-enumerate) devices and accept the values set in BARS and
other resources.
Now, I load the driver and .... Wham!
# modprobe e1000e
0:<6>e1000e: Intel(R) PRO/1000 Network Driver - 1.3.10-k2
0:<6>e1000e: Copyright(c) 1999 - 2011 Intel Corporation.
0:<6>e1000e 0000:04:00.0: Disabling ASPM L1
0:<7>PCI: Setting latency timer of device 0000:04:00.0 to 64
0:<3>e1000e 0000:04:00.0: MSI hi = 0x0, lo = 0xfcdc0000, data = 0x6
*******************************************************************************************
cpu_0 received a bus/cache error(warning)
*******************************************************************************************
CPU (XXX specific) registers dump: Node=0
COP2: TxMsgStatus = 0x00000000, MsgStatus1 = 0x0f0f0000
ICU: log0 = 0x00000000, log1 = 0x00000000, log2 = 0x00000000
LSU: log0 = 0x00000000, log1 = 0x0000003c88f64500
SCU: log0 = 0x00000000, log1 = 0x00000000, log2 = 0x00000000
TCU: reg0 = 0x00000000, reg1 = 0x00000000, reg2 = 0x00000000
NBU: reg0 = 0x00000000, reg1 = 0x00000000, reg2 = 0x00000000
DRAM_A: reg1 = 0x00000000, reg2 = 0x00000000
DRAM_B: reg1 = 0x00000000, reg2 = 0x00000000
DRAM_C: reg1 = 0x00000000, reg2 = 0x00000000
DRAM_D: reg1 = 0x00000000, reg2 = 0x00000000
CPU: epc = 0xffffffffc127e3fc, errorepc = 0xffffffffe00580c4, cacheerr = 0x00000000
*******************************************************************************************
LSU Cache Warning:
log0=0x0000000000000000, phyaddr=0x3c88f64500
Now that is a real problem.
After a couple of debug attempts, I could figure out that the first access
to PCI Mem configured in BAR0 is the culprit.
Obviously, we need to check whether BARs are setup correctly or not.
lspci comes to rescue here
lspci | grep 8086 /* This works because I am not on an intel system */
04:00.0 0200: 8086:10d3
~ # lspci -xxx -s 4:0.0
04:00.0 0200: 8086:10d3
00: 86 80 d3 10 07 00 10 00 00 00 00 02 08 00 00 00
10: 00 00 10 d0 00 00 20 d0 01 00 00 14 00 00 30 d0
20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 1f a0
30: 00 00 00 00 c8 00 00 00 00 00 00 00 58 01 00 00
40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a0: 11 00 04 00 03 00 00 00 03 20 00 00 00 00 00 00
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 01 d0 22 c8 00 20 00 14
d0: 05 e0 80 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 10 a0 01 00 c1 8c 00 00 10 28 10 00 11 1c 03 05
f0: 00 00 11 10 00 00 00 00 00 00 00 00 00 00 00 00
The highlighted area are the two BARs configured. The reason for crash is caused
by an access (otherwise legitimate -- not a bug in the driver) to one of these
areas (actually 0xd0100000 -- from the driver code). How do I know this? You look through the driver and you can make it out.
Why should we get a kernel crash while trying to access a memspace register? It can be explained like this. When an address is to be accessed, that address is put on the bus. Either DRAM controller should claim it (if it is within the DRAM range) or some other device controller should claim it, thus resolving the address. In this case, there were none to claim the address. Thus the NBU(North Bridge Unit) threw up an error.
Obviously that means this address' access is not set up correctly. We have to see how, when and why?
(I had set the PCI controller address range in the boot loader. Thus, I know 0xd0000000 is the base of PCI memspace and its size is 32M. And I had setup the NBU registers to redirect this memory range to PCI controllers. So, I don't have to worry about that part )
Armed with this knowledge, we need to figure out how access to this address is routed through PCIe switch from controller to the device.
lspci comes to rescue again.
~ # lspci -t -v #In tree structure, verbose
-[0000:00]-+-00.0 XXXX:1001
+-00.1 XXXX:1002
+-00.2 XXXX:1002
+-00.3 XXXX:1002
+-00.4 XXXX:1003
+-01.0-[0000:01-05]----00.0-[0000:02-05]--+-04.0-[0000:03]--
| +-05.0-[0000:04]----00.0 8 086:10d3
| \-08.0-[0000:05]--
+-01.1-[0000:06]--
+-01.2-[0000:07]--
+-01.3-[0000:08]--
What should we make out from this terse runic inscriptions?
The relevant facts here are that,
1. bus0,dev1,func0 is a pcie switch, so are d0f1, d0f2, and d0f3
2. b0d1f0 has a secondary bus (bus 1) and a subordinate bus(bus5).
why? because that is what 01.0-[0000:01-05] means
if you need raw config space data, try
~ # lspci -x -s 0:1.0
00:01.0 0604: XXXX:1004 (rev 10)
00: 4e 18 04 10 07 04 10 00 10 00 04 06 08 00 81 00
10: 00 00 00 00 00 00 00 00 00 01 05 00 01 f1 00 00
20: 00 10 30 d0 00 10 00 00 00 00 00 00 00 00 00 00
30: 00 14 0f 14 40 00 00 00 00 00 00 00 58 01 00 00
3. Following the above points, we can see that the memory access path is,
controller (0:1.0) -> switch(primary/upstream) (1:0.0) ->
switch (secondary/downstream) (2:5.0) -> device (4:0.0) for 0xd0100000 address.
Also, note that bus 2 is internally consumed by the switch.
This makes our task easy, we need to check only the above path to figure out
whether memory access works at all.
Again, back to lspci to check whether memspace is set correctly throughout this
path.
~ # lspci -xx -s 04:00.0
04:00.0 0200: 8086:10d3
00: 86 80 d3 10 07 00 10 00 00 00 00 02 08 00 00 00
10: 00 00 10 d0 00 00 20 d0 01 00 00 14 00 00 30 d0
20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 1f a0
30: 00 00 00 00 c8 00 00 00 00 00 00 00 58 01 00 00
Shows the device has two regions : one starting @ 0xd0100000 , another starting
@ 0xd0200000. We don't know the sizes of these regions, but for the time being
it does not matter.
For any access to these regions to pass through the switch, switch has to be
configured to claim these accesses. A switch can be programmed to do this by
filling in the correct range in memspace/mem limit registers at offset 0x20-0x23
~ # lspci -xx -s 01:00.0
01:00.0 0604: 10b5:8624 (rev bb)
00: b5 10 24 86 07 04 10 00 bb 00 04 06 08 00 01 00
10: 00 00 00 d0 00 00 00 00 01 02 05 00 f1 f1 00 00
20: 00 10 30 d0 01 10 01 00 00 00 00 00 00 00 00 00
30: 00 00 0f 14 40 00 00 00 00 00 00 00 58 01 00 00
Does not look good. Why? Pay attention to the highlighted area. This means that the range 0xd0300000 - 0x1000ffff is the memspace. Or is it ? No. according to the spec, if the mem limit is lower than base, then it should be considered disabled. So, effectively we have disabled the memspace behind the switch. No wonder we got a kernel crash.
Lets check if this behavior is consistent across all switch ports and controller (controller is also a switch, called root switch). Lets start with the controller.
~ # lspci -x -s 0:1.0
00:01.0 0604: XXXX:1004 (rev 10)
00: 4e 18 04 10 07 04 10 00 10 00 04 06 08 00 81 00
10: 00 00 00 00 00 00 00 00 00 01 05 00 01 f1 00 00
20: 00 d0 30 d0 00 10 00 00 00 00 00 00 00 00 00 00
30: 00 14 0f 14 40 00 00 00 00 00 00 00 58 01 00 00
Hmm... I guess you can see the discrepancy.
Now, to make sure the error has percolated all the way through,
~ # lspci -x -s 2:5.0
02:05.0 0604: 10b5:8624 (rev bb)
00: b5 10 24 86 07 04 10 00 bb 00 04 06 08 00 01 00
10: 00 00 00 00 00 00 00 00 02 04 04 00 f1 f1 00 00
20: 00 10 30 d0 01 10 01 00 00 00 00 00 00 00 00 00
30: 00 00 0f 14 40 00 00 00 00 00 00 00 58 01 00 00
~ # lspci -x -s 4:0.0
04:00.0 0200: 8086:10d3
00: 86 80 d3 10 07 00 10 00 00 00 00 02 08 00 00 00
10: 00 00 10 d0 00 00 20 d0 01 00 00 14 00 00 30 d0
20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 1f a0
30: 00 00 00 00 c8 00 00 00 00 00 00 00 58 01 00 00
This is completely weired. No wonder we had such a bad experience. Looks like
boot loader screwed up big time. So, lets drop to boot loader (in our case, it
is u-boot)
The correct minimum memory space access range should be as follows
0:1.0 : regiter at offset 0x20 (membase/memlimit) should contain something like
0xd010d020. Thus defining an address range on the secondary side of the root
switch to be 0xd0100000 - 0xd02fffff (at least)
1:0.0 : If this switch has not implemented a BAR for its internal PCI memory,
range 0xd0100000 - 0xd02fffff will solely be for seconday side of the switch.
2:5.0 : Requires 0xd0100000 - 0xd02fffff assigned to this device.
Now, lets drop to u-boot and manually overwrite the range
u-boot# pci write.l 0x20 0xd010d020 1.0.0
u-boot# pci write.l 0x20 0xd010d020 2.5.0
u-boot# pci write.l 0x20 0xd000d010 2.8.0
Boot up linux and load the driver.
~ # modprobe e1000e
0:<6>e1000e: Intel(R) PRO/1000 Network Driver - 1.3.10-k2
0:<6>e1000e: Copyright(c) 1999 - 2011 Intel Corporation.
0:<6>e1000e 0000:04:00.0: Disabling ASPM L0s
0:<7>PCI: Setting latency timer of device 0000:04:00.0 to 64
~ # ifconfig eth0 1.1.1.1 up
~ # ifconfig eth0 10.67.70.5 up
~ # ping 10.67.70.61
PING 10.67.70.61 (10.67.70.61): 56 data bytes
64 bytes from 10.67.70.61: seq=0 ttl=64 time=1.187 ms
64 bytes from 10.67.70.61: seq=1 ttl=64 time=0.137 ms
64 bytes from 10.67.70.61: seq=2 ttl=64 time=0.129 ms
That was a great experience to figure out the root cause of the crash. Now
update bootloader code to handle PCIe switch resource allocations correctly.
Thus, a problem manifested with reference to PCIe native hot-plug pointed to a PCI resource allocation issue which had to be eventually fixed in the boot loader in embedded system. Maybe, if we had used linux to re-initialize all PCI resources and re-enumerate it, this problem would have never occurred.
Anyway, now you, the reader, should be slightly more comfortable using lspci to troubleshoot some intricate PCI resource allocation issues :-)
our MIPS based board. Although we support native hotplug (by virtue of the
controller supporting native hotplug), the board did not have any hot-plug
support infrastructure : no attention button, no power control to slots ..etc.
So, we had to resort to a PLX chipset -- PEX 8624 and RDK board.
Thus, I started with an intel eepro (PCI ID - 8086:10d3) pcie single lane 1G ethernet
card. The driver has been available with linux and it is rock stable. After
compiling it as a module, (e1000e.ko) Linux was booted on the board.
Here is the topology
MIPS -> PCIe controller -> PEX 8624 -> eepro card
There are four 8x pcie controllers (yeah.. it can do 160Gbps sustained). Our
boot loader initializes the controllers and sets up BARs..etc. Linux is told not
to reconfigure (re-enumerate) devices and accept the values set in BARS and
other resources.
Now, I load the driver and .... Wham!
# modprobe e1000e
0:<6>e1000e: Intel(R) PRO/1000 Network Driver - 1.3.10-k2
0:<6>e1000e: Copyright(c) 1999 - 2011 Intel Corporation.
0:<6>e1000e 0000:04:00.0: Disabling ASPM L1
0:<7>PCI: Setting latency timer of device 0000:04:00.0 to 64
0:<3>e1000e 0000:04:00.0: MSI hi = 0x0, lo = 0xfcdc0000, data = 0x6
*******************************************************************************************
cpu_0 received a bus/cache error(warning)
*******************************************************************************************
CPU (XXX specific) registers dump: Node=0
COP2: TxMsgStatus = 0x00000000, MsgStatus1 = 0x0f0f0000
ICU: log0 = 0x00000000, log1 = 0x00000000, log2 = 0x00000000
LSU: log0 = 0x00000000, log1 = 0x0000003c88f64500
SCU: log0 = 0x00000000, log1 = 0x00000000, log2 = 0x00000000
TCU: reg0 = 0x00000000, reg1 = 0x00000000, reg2 = 0x00000000
NBU: reg0 = 0x00000000, reg1 = 0x00000000, reg2 = 0x00000000
DRAM_A: reg1 = 0x00000000, reg2 = 0x00000000
DRAM_B: reg1 = 0x00000000, reg2 = 0x00000000
DRAM_C: reg1 = 0x00000000, reg2 = 0x00000000
DRAM_D: reg1 = 0x00000000, reg2 = 0x00000000
CPU: epc = 0xffffffffc127e3fc, errorepc = 0xffffffffe00580c4, cacheerr = 0x00000000
*******************************************************************************************
LSU Cache Warning:
log0=0x0000000000000000, phyaddr=0x3c88f64500
Now that is a real problem.
After a couple of debug attempts, I could figure out that the first access
to PCI Mem configured in BAR0 is the culprit.
Obviously, we need to check whether BARs are setup correctly or not.
lspci comes to rescue here
lspci | grep 8086 /* This works because I am not on an intel system */
04:00.0 0200: 8086:10d3
~ # lspci -xxx -s 4:0.0
04:00.0 0200: 8086:10d3
00: 86 80 d3 10 07 00 10 00 00 00 00 02 08 00 00 00
10: 00 00 10 d0 00 00 20 d0 01 00 00 14 00 00 30 d0
20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 1f a0
30: 00 00 00 00 c8 00 00 00 00 00 00 00 58 01 00 00
40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a0: 11 00 04 00 03 00 00 00 03 20 00 00 00 00 00 00
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 01 d0 22 c8 00 20 00 14
d0: 05 e0 80 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 10 a0 01 00 c1 8c 00 00 10 28 10 00 11 1c 03 05
f0: 00 00 11 10 00 00 00 00 00 00 00 00 00 00 00 00
The highlighted area are the two BARs configured. The reason for crash is caused
by an access (otherwise legitimate -- not a bug in the driver) to one of these
areas (actually 0xd0100000 -- from the driver code). How do I know this? You look through the driver and you can make it out.
Why should we get a kernel crash while trying to access a memspace register? It can be explained like this. When an address is to be accessed, that address is put on the bus. Either DRAM controller should claim it (if it is within the DRAM range) or some other device controller should claim it, thus resolving the address. In this case, there were none to claim the address. Thus the NBU(North Bridge Unit) threw up an error.
Obviously that means this address' access is not set up correctly. We have to see how, when and why?
(I had set the PCI controller address range in the boot loader. Thus, I know 0xd0000000 is the base of PCI memspace and its size is 32M. And I had setup the NBU registers to redirect this memory range to PCI controllers. So, I don't have to worry about that part )
Armed with this knowledge, we need to figure out how access to this address is routed through PCIe switch from controller to the device.
lspci comes to rescue again.
~ # lspci -t -v #In tree structure, verbose
-[0000:00]-+-00.0 XXXX:1001
+-00.1 XXXX:1002
+-00.2 XXXX:1002
+-00.3 XXXX:1002
+-00.4 XXXX:1003
+-01.0-[0000:01-05]----00.0-[0000:02-05]--+-04.0-[0000:03]--
| +-05.0-[0000:04]----00.0 8 086:10d3
| \-08.0-[0000:05]--
+-01.1-[0000:06]--
+-01.2-[0000:07]--
+-01.3-[0000:08]--
What should we make out from this terse runic inscriptions?
The relevant facts here are that,
1. bus0,dev1,func0 is a pcie switch, so are d0f1, d0f2, and d0f3
2. b0d1f0 has a secondary bus (bus 1) and a subordinate bus(bus5).
why? because that is what 01.0-[0000:01-05] means
if you need raw config space data, try
~ # lspci -x -s 0:1.0
00:01.0 0604: XXXX:1004 (rev 10)
00: 4e 18 04 10 07 04 10 00 10 00 04 06 08 00 81 00
10: 00 00 00 00 00 00 00 00 00 01 05 00 01 f1 00 00
20: 00 10 30 d0 00 10 00 00 00 00 00 00 00 00 00 00
30: 00 14 0f 14 40 00 00 00 00 00 00 00 58 01 00 00
3. Following the above points, we can see that the memory access path is,
controller (0:1.0) -> switch(primary/upstream) (1:0.0) ->
switch (secondary/downstream) (2:5.0) -> device (4:0.0) for 0xd0100000 address.
Also, note that bus 2 is internally consumed by the switch.
This makes our task easy, we need to check only the above path to figure out
whether memory access works at all.
Again, back to lspci to check whether memspace is set correctly throughout this
path.
~ # lspci -xx -s 04:00.0
04:00.0 0200: 8086:10d3
00: 86 80 d3 10 07 00 10 00 00 00 00 02 08 00 00 00
10: 00 00 10 d0 00 00 20 d0 01 00 00 14 00 00 30 d0
20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 1f a0
30: 00 00 00 00 c8 00 00 00 00 00 00 00 58 01 00 00
Shows the device has two regions : one starting @ 0xd0100000 , another starting
@ 0xd0200000. We don't know the sizes of these regions, but for the time being
it does not matter.
For any access to these regions to pass through the switch, switch has to be
configured to claim these accesses. A switch can be programmed to do this by
filling in the correct range in memspace/mem limit registers at offset 0x20-0x23
~ # lspci -xx -s 01:00.0
01:00.0 0604: 10b5:8624 (rev bb)
00: b5 10 24 86 07 04 10 00 bb 00 04 06 08 00 01 00
10: 00 00 00 d0 00 00 00 00 01 02 05 00 f1 f1 00 00
20: 00 10 30 d0 01 10 01 00 00 00 00 00 00 00 00 00
30: 00 00 0f 14 40 00 00 00 00 00 00 00 58 01 00 00
Does not look good. Why? Pay attention to the highlighted area. This means that the range 0xd0300000 - 0x1000ffff is the memspace. Or is it ? No. according to the spec, if the mem limit is lower than base, then it should be considered disabled. So, effectively we have disabled the memspace behind the switch. No wonder we got a kernel crash.
Lets check if this behavior is consistent across all switch ports and controller (controller is also a switch, called root switch). Lets start with the controller.
~ # lspci -x -s 0:1.0
00:01.0 0604: XXXX:1004 (rev 10)
00: 4e 18 04 10 07 04 10 00 10 00 04 06 08 00 81 00
10: 00 00 00 00 00 00 00 00 00 01 05 00 01 f1 00 00
20: 00 d0 30 d0 00 10 00 00 00 00 00 00 00 00 00 00
30: 00 14 0f 14 40 00 00 00 00 00 00 00 58 01 00 00
Hmm... I guess you can see the discrepancy.
Now, to make sure the error has percolated all the way through,
~ # lspci -x -s 2:5.0
02:05.0 0604: 10b5:8624 (rev bb)
00: b5 10 24 86 07 04 10 00 bb 00 04 06 08 00 01 00
10: 00 00 00 00 00 00 00 00 02 04 04 00 f1 f1 00 00
20: 00 10 30 d0 01 10 01 00 00 00 00 00 00 00 00 00
30: 00 00 0f 14 40 00 00 00 00 00 00 00 58 01 00 00
~ # lspci -x -s 4:0.0
04:00.0 0200: 8086:10d3
00: 86 80 d3 10 07 00 10 00 00 00 00 02 08 00 00 00
10: 00 00 10 d0 00 00 20 d0 01 00 00 14 00 00 30 d0
20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 1f a0
30: 00 00 00 00 c8 00 00 00 00 00 00 00 58 01 00 00
This is completely weired. No wonder we had such a bad experience. Looks like
boot loader screwed up big time. So, lets drop to boot loader (in our case, it
is u-boot)
The correct minimum memory space access range should be as follows
0:1.0 : regiter at offset 0x20 (membase/memlimit) should contain something like
0xd010d020. Thus defining an address range on the secondary side of the root
switch to be 0xd0100000 - 0xd02fffff (at least)
1:0.0 : If this switch has not implemented a BAR for its internal PCI memory,
range 0xd0100000 - 0xd02fffff will solely be for seconday side of the switch.
2:5.0 : Requires 0xd0100000 - 0xd02fffff assigned to this device.
Now, lets drop to u-boot and manually overwrite the range
u-boot# pci write.l 0x20 0xd010d020 1.0.0
u-boot# pci write.l 0x20 0xd010d020 2.5.0
u-boot# pci write.l 0x20 0xd000d010 2.8.0
Boot up linux and load the driver.
~ # modprobe e1000e
0:<6>e1000e: Intel(R) PRO/1000 Network Driver - 1.3.10-k2
0:<6>e1000e: Copyright(c) 1999 - 2011 Intel Corporation.
0:<6>e1000e 0000:04:00.0: Disabling ASPM L0s
0:<7>PCI: Setting latency timer of device 0000:04:00.0 to 64
~ # ifconfig eth0 1.1.1.1 up
~ # ifconfig eth0 10.67.70.5 up
~ # ping 10.67.70.61
PING 10.67.70.61 (10.67.70.61): 56 data bytes
64 bytes from 10.67.70.61: seq=0 ttl=64 time=1.187 ms
64 bytes from 10.67.70.61: seq=1 ttl=64 time=0.137 ms
64 bytes from 10.67.70.61: seq=2 ttl=64 time=0.129 ms
That was a great experience to figure out the root cause of the crash. Now
update bootloader code to handle PCIe switch resource allocations correctly.
Thus, a problem manifested with reference to PCIe native hot-plug pointed to a PCI resource allocation issue which had to be eventually fixed in the boot loader in embedded system. Maybe, if we had used linux to re-initialize all PCI resources and re-enumerate it, this problem would have never occurred.
Anyway, now you, the reader, should be slightly more comfortable using lspci to troubleshoot some intricate PCI resource allocation issues :-)
Subscribe to:
Posts (Atom)