Personal tools

OpenLaszlo Release Tests

From OpenLaszlo

This document describes the tests required to certify a build for release. This document should be copied and updated by the test team as each step is accomplished. Link to detailed reports as they are generated. Many of these steps are assigned to different departments/roles. Those roles are designated here where assigned. The steps here are summarize in the OpenLaszlo Release Tests Master Check List UPDATE: As of the OL4 release cycle, these steps are out of date. An updated version of the Release Process is in progress.

  1. Preparation:
    1. Update build.properties file to match the release naming/versioning. Verify "Production" release
    2. Update the docs/release-notes.html.
    3. Update the docs/changelog.html.
    4. Update the README and README.core files. (and README.src when it exists)
    5. Check the top level build.xml file to make sure it has the files in its manifests.
    6. Engineering/Webmaster: Update the contents of the lps/docs/install with Antun/Sundman.
    7. Create a p4 branch spec for this release and integrate to create a release branch.
    8. If there is a release branch, integrate trunk to the release branch.
    9. Run a build from the release branch. See the automated build doc. This will set the build number for the release and initialize the spin number to 0.
    10. The builds will be at:
  2. Rinse: (This section to be reviewed/updated by MD)
    1. Read the complete build logs, look for compilation and build errors.
    2. Verify all fixed bugs
    3. Build Sanity Checks
      • grep for @VERSIONID@ in package
      • grep for @BUILD@ in package
      • grep for @LPS_NAME@
      • grep docs/guide/preface.html for FIXME
      • (for a production build) grep docs/guide/preface.html for TODO (this should be present in beta builds, absent in release)
    4. Version sanity checks
      • Use the lps/admin/console.lzx example to
        • check the version
          • check the detailed lps log file for the correct:
          • Build date
          • Build id
      • Smoke checks
      • Install/re-install testing
        • OSX
        • Linux
        • WindowsXP
        • Solaris 10
        • Windows Server 2003
      • Regression testing
        • Check the detailed lps log file ($LPS_HOME/WEB-INF/lps/work/logs/lps.log) for any ERROR or WARN messages.
        • Detailed regression testing
        • Client compatibility (OS, browser, flash-plugin) Testplan
          • Linux 2.6 FF1.0 FP7
          • Windows XP IE6 FP7
          • Windows XP IE6 FP8
          • Windows XP FireFox FP6
          • Windows XP FireFox FP7
          • Windows 2K IE6 FP7
          • Windows 2K FireFox FP7
          • OSX 10.4 Firefox FP8
          • OSX 10.4 Safari FP8
          • OSX 10.3 Safari FP7
          • OSX 10.3 FireFox FP7
          • test weather on the ipaq
        • Server tests
          • compatibility
          • availability
          • robustness
          • scalability
        • Link check the webapp
        • Performance testing:
          • Data/results to be stored in \bacall\laszlo\Intranet\engineering\LPS-Release-Criteria (these bits need to be moved to p4 at some point). Note: this work has often occurred post release acceptance below. This is not ideal for a number of reasons.
          • Client
            • Start-up time
            • Compiler
            • First compile (hello.lzx) after starting server
            • Compile dashboard (not 1st compile)
            • Compile calendar (not 1st compile)
            • Re-compile dashboard (one line change)
            • Re-compile calendar (one line change)
          • Server
            • General performance
            • App serving
            • Data
            • Media
            • Connections
            • Pot Store
        • Search for TODO/XXXs in all example, demos, tutorials, docs.
        • Verify copyrights in all examples, demos, docs, tutorials.
        • Verify changelog and release notes are complete and correct.
        • Verify no hard tabs in shipping example source code.
        • Check to make sure all the requisite credits are in $INSTALLDIR/Credits.
        • Spell check docs and tutorials.
        • List our all files in one of the installers and make sure we're not shipping anything that doesn't belong
  3. Repeat: If there are any problems... re-Lather and repeat. If there are changes required post-freeze, check them into the release branch in perforce (per the change control process) and either re-spin or re-build. A spin only pulls files that have changed; a re-build pulls the entire tree again. Re-building will reset the build number; re-spinning will leave the build number the same and increment the spin. See the automated build doc for details. Note: the respin process is currently broken; use full rebuilds for now.
  4. Post-Acceptance
    1. The release is all spickety Clean and has been accepted!
    2. Engineering: Lock the perforce label for the release.
    3. Engineering: back-port any changes in the release branch back up to the trunk. Announce that the trunk is open.
    4. QA/Antun: make sure bugs marked TECHNOTE are published to the external faq.
    5. Yossie/Engineering: Update //depot/build/index.php to hardcode the release version/build id.
    6. QA: Mark VERIFIED bugs as CLOSED.
    7. Engineering: Create a Bill of Materials (bom) for the release.
      1. Create a bom directory under //depot/boms with the full version and build id
      2. Windows, Mac, Linux, Core installers/packages from above.
      3. Source Distro from linux-builder (zip and tar.gz)
      4. Create MD5 Sums for all packages openssl dgst -md5 {filename} > filename.sum
      5. Create torrent files for all packages btmakemetafile.py {filename} http://torrent.laszlosystems.com:6969/announce
      6. Check in MD5 Sums and torrents
      7. evidence.txt file (ls -l * > evidence.txt)
      8. Create bom.txt, including date, version, build-id and all parts including documents and related web-site bits.
      9. Submit bom to perforce.
      10. Update performance metrics docs and add to bom in perforce.
    8. Check with marketing on need to a version of LPS with demos installed. Optionally create, build, and minimally qa a new demo branch and installer(s).
    9. Schedule a Release Meeting and produce the following
      1. The release plan/criteria set out.
      2. bom.txt (SKU list of all deliverables and % completion of each)
      3. Evidence of all deliverables
      4. Defect list with all non-closed P1\ufffds detailed in rank order of impact
      5. Defect chart of arrival rates and fix rates, etc.
      6. Performance metrics with latest measurements (rank order the ones that are off the most)
      7. Written QA assessment (around 1-2 paragraphs) and written recommendation
      8. Copy of release notes
      9. Written Waiver Request (this is what the release committee is being asked to approve)
    10. update //depot/www/var/www/html/developers/download/requirements/index.php to reflect current requirements. (Coordinate with antun before check in)
    11. Upload the release do the downloads section of our website. Antun/yossie have the latest on how this is done.
    12. Copy release packages and torrents to BitTorrent Server
      1. scp [files] tracker@torrent:files/ (password hint: leet potato)
      2. scp [torrents] tracker@torrent:files/
    13. Upgrade the site to be running the release.
    14. We may not even need to upgrade the site. But we should.
      1. apache
      2. jetty
      3. lps configuration file (license, lps.properties, log4j.properties and so on)
      4. Details from Antun/yossie.
    15. Link check the updated parts of the web site.
    16. Update site links, demos, docs as well
Retrieved from "http://wiki.openlaszlo.org/OpenLaszlo_Release_Tests"