Anna Gray

Anna Gray

Broadcast Systems Engineer : design and operations

https://uk.linkedin.com/in/anna-gray-81342b31
http://iweua.com
https://www.strava.com/athletes/agiweua
w e b 2 0 at i w e u a dot c o m

Location U.K..... mostly....for now, at least.

Activity

  • Yield a little bit like C's Static??

  • My only consideration : should I make one function taking radius and height, which returned two values, Area and Volume, or two functions, one returning Area and one returning Volume.

    I chose two functions for function clarity and simplicity ( to my C programmer's mind )

    round ( CylinderArea (radius,height), 2 )
    returns 471.24
    round (...

  • Anna Gray made a comment

    Python... default argument values : genius!
    It will tend to push complexity into the functions which need to be thoroughly tested. The more complex the function with the more parameters it can take, the harder it is to test under all conditions.

  • I have been learning Python for about a year, never came across *args / **kwargs and was weirded out a little as I thought they were pointers or something. Some of the ( many ) mysteries of Python are evaporating quite quickly with this course.

  • Anna Gray made a comment

    Python is just so cool !
    I guess it can get you into lots of trouble with it too!
    ( old C coder talking )

  • Run python_argparse_ex1.py with the argument ' -h' ( h = usually for 'help' )
    >python python_argparse_ex1.py -h

    usage: python_argparse_ex1.py [-h]

    optional arguments:
    -h, --help show this help message and exit

  • The ipnb files are Jupyter Notebooks files. They are the course work files seen in the following videos that you can replicate on your laptop in a Browser or by installing Jupyter labs on your local machine : HOW?

    To try quickly in your browser

    goto : https://jupyter.org/try

    A new page will open : https://hub.mybinder.turing.ac.uk/....

    There is...

  • Yep : so you can type
    python python_argv_ex1.py -test

    The output will be something like
    ['python_argv_ex1.py', '-test']

    Now we can check if args are passed when running the script
    if sys.argv[1] == "-test":
    print("TEST")

  • Azure is a huge environment, isn't it?
    I hadn't quite realised until now.

  • Five crucial elements
    1. Azure identity services : Let the right people in, the wrong ones out
    2. Security tools and features : Security is vital
    3. Privacy, compliance and data protection standards : Best practice
    4. Secure network : Essential though most N/W traffic now secure
    5. Monitoring and reporting : To see what is and isn't working

  • GDPR was / is a EU directive with the UK now retaining the legislation and in many cases adopting a more rigorous approach. I don't think anyone wants to go near trying to unpick it.

    To untangle data and geo locate it for many international companies is well nigh impossible, and who would / what authority in reality would / could try to audit that, and...

  • Anna Gray made a comment

    I note that products such as Solarwinds and Zabbix monitoring instances can be built on Azure as 3rd party monitoring.
    These can report on bandwidth, memory, cpu usage, hard disk capacity, load balancing issues etc, vital useful on high load or high traffic peak, real time systems.

  • The use by default of password complexity rules, coupled with the widespread adoption of password managers ( Lastpass/Keepass/Browser ) has reduced the compromise of accounts significantly. Limiting access to accounts by the use of IP whitelist rules also reduces the compromise surface. That is not to say it still doesn't happen, just that it shouldn't in...

  • 30 Nov 21 : got it working after some time
    https://docs.microsoft.com/en-us/cli/azure/install-azure-cli

    Install the CLI on Windows and Linux ( I chose Ubuntu CLI )
    I chose : Option 1: Install with one command
    then to run the program
    $ sudo az login
    You will be asked to navigate to
    https://login.microsoftonline.com/common/oauth2/deviceauth
    and...

  • @SörenLairdSörries @DaveGunn

    The term Container in software and IT is a little nebulous, but they are essentially a ( ideally single ) software process running / depending / utilising a larger operating system superstructure.

    Thus multiple containers can run on a single Windows/Linux machine ( eg see Docker ) Much like individual shipping containers...

  • Really good introduction to Azure and demonstrates just how straightforward 'spinning up' an online resource can be

  • Top left of Azure control panel screen
    Home>VMName>Networking>Add inbound port rule >
    Source : Any
    Source Port : * (a http request can come from a large range of ports )
    Destination : Any ( We can send traffic to any public IP Address )
    Service : HTTP : Port 80
    Priority : default ( 330
    Name : Port_HTTP

  • The ( currently ) single Azure data centre for the whole continent of Africa is located in South Africa. You would be right to think that some provision should be held off the west coast catering to Ghana,Nigeria, Cameroon and Gabon for example. Currently no.

    I assume there just isn't the demand,that current West African country network infrastructure...

  • Samuel, you raise an excellent point where dedicated machines in your industry, healthcare, manufacturing etc have high functionality, working, stable software, often running on legacy operating systems ( windows / embedded linux etc ) that can't be put 'in the cloud'.

  • The division of visual diagrams into 4 key categories,
    Comparison,
    Distribution,
    Composition
    Relationship
    is very useful as an immediate guide on which type of diagram is optimal to display the data and conclusions you wish to draw from the data. thank you

  • The Promo of 50% doesn't work, or is way to generous. Promo days resulted in similar sales quantity ( coffees served / day ) as other days but half the revenue, they didn't increase number of coffees sold ( significantly )
    Shop doesn't need 8 staff, especially Mon-Wed
    Little correlation between Avg Temperate and Hot/Cold coffee sales
    Busiest days are Friday...

  • We moved on to Energy futures?

  • Anna Gray made a comment

    Wow : I feel I learnt a LOT from ZERO to running basic SQL queries, something I have been meaning to learn for about 20 years, and I had to think it through, and only scratching the surface.

  • select customer.firstname,customer.lastname,sum(invoice.total) from customer
    inner join Invoice
    where customer.customerid = invoice.customerid
    and ( customer.customerid = 1
    or customer.customerid = 6
    or customer.customerid = 59
    )
    group by customer.customerid
    order by -sum(invoice.total); /* The negative for descending order...

  • I can see how a real SQL query could become quite complex, quite quickly.

  • Anna Gray made a comment

    Got stuck here...for a couple of days.. moved on. Subqueries and Joins from multiple tables needs more work and more documentation....

  • Anna Gray made a comment

    Just spent a really engaging few days ( about 5 ) getting to grips with Sqlite, filters, code and the sheer power of it. I am very impressed while realising I am just scratching the surface of the power of this tool.

    I am fortunate that I code ( C / Python ) also so can see parallels / different ways of achieving tasks, and do appreciate that SQL provides...

  • So a personal learning experience. I spent about 4/5 days late October writing some Python code to filter the flights.csv file and was successful

    Now I have spent about 4 days learning SQLite and have two identical CSV files of filtered departures for LAX, days 5 ( time >= 1700 ) /6/ 7 ( time < 1200 )
    I have learnt a lot about big files and the power of...

  • Anna Gray made a comment

    select firstname,lastname from customer where customerid = 42;

    customer id is an INTEGER, match can be with integer, no quotes required

    select employeeid,firstname,lastname from employee where address like '%77%';

    address is NVARCHAR(70), thus LIKE search term ( 77 ) is in quotes

    sqlite> select count(customerid) from customer where postalcode is...

  • Fascinating to be working on real but sufficiently complex dataset

    @Richard R : Good spot '<=' is correct, =< does not work

  • A potential for confusion is that the download from github gives two sqlite files, a .sql and a .sqlite.
    The chinook_sqlite.sql is human readable using a text editor ( sublime text etc ) . The .sql file is in effect a .dump of the database file

    In SQLite you can read the sql or open the sqlite

    sqlite> .read Chinook_Sqlite.sql
    or
    sqlite> .open...

  • @AndrewB @BolajiAhmed :
    what extent this actually happens in the 'real world'.
    Amost never : data would be added directly to the SQL DB, only by DB maintenance or analysis IT devs manually in this fashion.
    The majority of manual data entry into is done via Web interfaces which allow / provide software data error, validity and form checking ( formats of...

  • @AndrewB : I wonder to what extent this actually happens in the 'real world'.

  • sqlite> .mode csv
    sqlite> .import flights.csv flights
    sqlite> .tables
    flights
    sqlite> select count(year) from flights;
    5819079

    Cool : took a couple of days working around SQLite to get to this point, still getting the hang of adding a ; to the end of NON DOT commands, and importantly NOT adding a Semicolon to the end of DOT commands as you end up with...

  • SQLite is BIG!

  • I am going to have to reread this a few times I think!
    This is important and fundamental.

  • Why would it make more sense to use a separate integer value for a primary key
    Format constraints for PK and error checking capability prior to Query of PK
    Data security : PK not based on P.I.I. ( eg name / DOB )..GDPR
    Immutable : The key never has to change, other aspects of the record can be updated..name...address..phone....etc
    Zero potential for...

  • There is a LOT of data out there to collect, with the 'data' and the complexity of it evolving all the time.
    More complex models will be required to capture and link related data together ( I guess ! ) to allow relationships between data points to be found, mapped and information derived.

  • Anna Gray made a comment

    Scatter plot for this data is pure noise with zero correlation between height to weight, zero, even when boxed off to 5/10 cm intervals.
    This can't be a data set for sports anything, and unlikely Olympian women who would typically come in around 55kg for a 165cm female athlete ( eg Laura Kenny ).
    Even if this data was for a specialised sport that is...

  • Great to see the FT open-sourcing its data presentation template as best practice

  • Seeing the wood from the trees can be hard : visualising information in more and more interesting ways has become very useful, interpreting the data remains something of an art.

  • Anna Gray made a comment

    Didn't know pivot tables before this evening

  • I can see that with numbers as easy to read as a 24 hour time, written in 2/3 digit numbers 730, not 0730, or a month in text ( January, February etc ) converting this into numbers which are easily sorted can be hard in excel.

    Also calculation in base 7/12/50 or 24 ( 0830 - 0930 is not 100 but 60 minutes ) for example

  • Anna Gray made a comment

    Enjoyed the Excel left/right exercise, long time user of spreadsheets with quite complex formulas ( references, offsets, rolling averages etc ) but not used those

  • From sample-flights.csv
    I have 213 flights departure LAX all days 1-7
    88 flights departure days 5/6/7 LAX
    47 flights LAX between
    Day 5 departure after 1700 ( Tail N8654B )
    and departure
    Day 7 before 1200 midday ( tail N364AA)
    sorted by departure time

  • Wow, that was a learning experience
    =VALUE(RIGHT(B2,4)) for the year in numbers
    =LEFT(B2, SEARCH(" ",B2)) for the Month in Text
    =LEFT(B2,SEARCH(",",B2)-1) gave me eg April 23 from April23, 1983
    =VALUE(RIGHT(C2,(LEN(C2)- SEARCH(" ",C2) ))) gave me 23 from April 23
    or
    9 from July 9
    As you had to calculate the length of string between the Space and the end...

  • @PeterTurnbull @RichardR

    They made the same mistake in the last course confusing Variance Population with Variance sample

    Variance Population = 240 / 9 = 26.66 ( we all agree )
    Sum of (Mean - vale ) ^ 2 = 240
    n Population = 9

    Variance sample which has been quoted as the answer
    = 240 / ( Population- 1 ) = 240 / 8 = 30

  • I think we are looking for patterns here and outliers. Is there a destination, airline or time of day that is 'problematic' with extra delayed or cancelled flights?

  • An hour lost on this : is it me or is there an error and confusion regarding the sample set and quoted Variance calculations

    For data
    53,72,75,33,30,76,22,86,59,56,81,49,61,20,94
    Sum = 867
    Mean = 57.8
    Sum of square ( xi - 57.8 ) ^2 = 7686.4

    n = 15
    n-1 = 14

    7686.4 / 15 = 512.43 ( Variance : Population )
    7686.4 / 14 = 549.03 ( Variance : Sample...