texts: Scraping a webpage copying with the logic of scrapy

texts:
Today, while coming across a tutorial made by ScrapingHub on Scrapy about how it usually deals with a webpage while scraping it's content. I could see that the same logic applied in Scrapy can be applied in regular method in python.

So, I tried to make one. Here is what I did. My scraper opens a webpage, parses the 10 tags from it's right sided area then tracks down each tags and then again follows its pagination and parses the whole content. Hope I did it the right way. Here is the code:

import requests ; from lxml import html

core_link = "http://quotes.toscrape.com/"

def quotes_scraper(base_link):
    response = requests.get(base_link)
    tree = html.fromstring(response.text)
    for titles in tree.cssselect("span.tag-item a.tag"):
        processing_docs(core_link + titles.attrib['href'])

def processing_docs(base_link):
    response = requests.get(base_link).text
    root = html.fromstring(response)
    for soups in root.cssselect("div.quote"):
        quote = soups.cssselect("span.text")[0].text
        author = soups.cssselect("small.author")[0].text
        print(quote, author)

    next_page = root.cssselect("li.next a")[0].attrib['href'] if root.cssselect("li.next a") else ""
    if next_page:
        page_link = core_link + next_page
        processing_docs(page_link)

quotes_scraper(core_link)


texts:
My original code is here. I made some changes to it based on your suggestions, but then I lost my flash drive and had to redo it. I have a suspicion that my second version was better than this one, so could you all help me out one more time?

import os, shutil, zipfile

def subtractname(path):
    newpath = str()
    path = path.split("\\")
    path.pop()
    for x in range(len(path) - 1):
        newpath += (path[x] + "\\")
    newpath += path[len(path) - 1]
    return newpath

def pathtoname(path):
    path = path.split("\\")
    newpath = path[len(path) - 1]
    newpath = newpath[:-4]
    return(newpath)

def makepath(path):
    try:
        os.makedirs(path)
    except FileExistsError:
        shutil.rmtree(path)
        os.makedirs(path)

def getscds(targetfolder):
    global scdlist
    filelist = os.listdir(targetfolder)
    scdlist = []
    for file in filelist:
        if '.scd' in file:
            scdlist.append(os.path.join(targetfolder, file))
    return scdlist

def unzip(file, target):
    modscd = zipfile.ZipFile(file)
    makepath(target)
    modscd.extractall(target)

def checkdirs(target):
    isfile = 0
    targetlist = os.listdir(target)
    for files in targetlist:
        if not os.path.isfile(files):
            isfile = isfile + 1
    if isfile == len(os.listdir(target)):
        return False
    else:
        return True

def install(folderlist):
    newpathlist = []
    for folder in folderlist:
        pathlist = getscds(folder)
        for path in pathlist:
           newpath = os.path.join(subtractname(path), ("f_" + pathtoname(path)))
           newpathlist.append(newpath)
           unzip(path, newpath)
           if checkdirs(newpath):
            shutil.copy(path, r"C:\Program Files (x86)\Steam\steamapps\common\supreme commander 2\gamedata")
            newpathlist.remove(newpath)
    return newpathlist

a = input()
b = [a]
install(install(install(b)))

A note: this code SHOULD actually work if you run it, but I haven't incorporated error handling yet.

I know that the last three lines can be condensed into one, but aside from that is there anything that could be improved here? I'm not really looking for naming conventions or comments, but I'm fine with it if you want to throw them in along with some more practical suggestions.

EDIT: I just tried running this on a friend's computer, and it may not work after all. Ideas?


texts: Using regex to replace strange characters

texts:
After importing some products from a csv strange characters have shown up on the page and it would be too much work to manually go to each product and remove them so I made this script to deploy on that product page and remove them.

$(function() {
    var p_desc = $(".rte").html();
    var re = /\?ÕÌ_|Š|š|Ž|ž|À|Á|Â|Ã|Ä|Å|Æ|Ç|È|É|Ê|Ë|Ì|Í|Î|Ï|Ñ|Ò|Ó|Ô|Õ|Ö|Ø|Ù|Ú|Û|Ü|Ý|Þ|ß|à|á|â|ã|ä|å|æ|ç|è|é|ê|ë|ì|í|î|ï|ð|ñ|ò|ó|ô|õ|ö|ø|ù|ú|û|ý|þ|ÿ|_Œ‚|__|_/g;
    var result = p_desc.replace(re, ' ');
    var new_p_desc = result.replace(/[^\x00-\x7F]/g, "").replace(/\?/g, '');
    $(".rte").html(new_p_desc);
});

My script is working fine but not sure if it could be made better. Was this the best way to go about it?


texts: Analysis describing the combination of cards

texts:
In this class, the order of the last 3 statements in the constructor is important. If you call OrderedCards = OrderCards(cards, wildcardsCount); last, it will crash with NRE.

Is this a bad idea? If so, how should I fix it?

/// <summary>
/// The abstract result of analysis describing the combination of cards
/// </summary>
public abstract class TichuTrick
{

    public List<CardData> OrderedCards { get; private set; }
    public readonly int CardCount;
    public readonly int TrickValue;
    public readonly int HighestCardValue;
    public readonly int WildcardsCount;

    /// <summary>
    /// Constructs a response of analysis
    /// </summary>
    /// <param name="cards">This class works only with 1 wildcard in each trick.</param>
    internal TichuTrick(List<CardData> cards, int cardCount, int wildcardsCount)
    {
        if (wildcardsCount > 1)
        {
            throw new System.ArgumentException("More than one wildcards support, is not yet implemented", nameof(wildcardsCount));
        }
        CardCount = cardCount;
        OrderedCards = OrderCards(cards, wildcardsCount);
        TrickValue = CalculateTrickValue();
        HighestCardValue = CalculateHighestCardValue();
    }

    protected virtual List<CardData> OrderCards(List<CardData> cards, int WildcardCount)
    {
        return cards.OrderBy(x => x.NumericalValue).ToList().CopyList();
    }

    protected virtual int CalculateTrickValue()
    {
        return CalculateHighestCardValue();
    }

    protected virtual int CalculateHighestCardValue()
    {
        return OrderedCards.Last().NumericalValue;
    }

}


texts: Three ways to sum the values of letters in words

texts:
I've created these 3 functions to count the value of letters in a word.

All these functions are functionally the same.

Apparently List comprehensions (count_word3) are usually seen as the most pythonic in these situations. But to me it is the least clear of the 3 examples.

Have I stumbled across something that is so simple that lambdas are acceptable here or is "the right thing to do" to use the list comprehension?

Or is there something better?

def count_word1(w):
    def count(i, a):
        return i + ord(a) - 64
    return reduce(count, w, 0)

def count_word2(w):
    return reduce(lambda a,b : a + ord(b) - 64, w, 0)

def count_word3(w):
    vals = [ord(c) - 64 for c in w]
    return sum(vals)


texts: functional javascript - how could I generalize this code that correlates parallel async requests with their results?

texts:
/**
 * takes a list of componentIDs to load, relative to componentRoot
 * returns a promise to the map of (ComponentID -> componentCfg)
 */
function asyncLoadComponents (componentRoot, components) {

    var componentCfgs = {};

    function asyncLoadComponentCfg(component) {
        var url = _.sprintf("%s/%s", componentRoot, component);
        var promise = util.getJSON(url);
        promise.done(function(data) {
            componentCfgs[component] = data;
        });
        return promise;
    }

    var promises = _.map(components, asyncLoadComponentCfg);
    var flattenedPromise = $.when.apply(null, promises);
    var componentCfgPromise = flattenedPromise.pipe(function() {
        // componentCfgs is loaded now
        return $.Deferred().resolve(componentCfgs).promise();
    });

    return componentCfgPromise;
}

var locale = 'en-US';
var componentRoot = '/api/components';
var components = ['facets', 'header', 'DocumentList'];
function buildDocumentListPage(locale, componentCfgs) { /* code goes here */ }

$.when(asyncLoadComponents(componentRoot, components)).done(function(componentCfgs) {
    buildDocumentListPage(locale, componentCfgs)
});


texts: JWT authentication between Django and ReactJS

texts:
I am currently using Django (2.1) to build an API, and I have added djangorestframework-jwt to manage JWT.

Here is the configuration:

JWT_AUTH = {
    'JWT_SECRET_KEY': SECRET_KEY,
    'JWT_VERIFY': True,
    'JWT_VERIFY_EXPIRATION': True,
    'JWT_EXPIRATION_DELTA': datetime.timedelta(days=14),
    'JWT_ALLOW_REFRESH': True,
    'JWT_REFRESH_EXPIRATION_DELTA': datetime.timedelta(days=7),
    'JWT_AUTH_HEADER_PREFIX': 'Bearer',
}

and the endpoints:

urlpatterns = [
    path('auth/get-token/', obtain_jwt_token),
    path('auth/refresh-token/', refresh_jwt_token),
]

The client is built with ReactJS. I use an axios instance as a client to communicate with the API. This instance is created that way :

import axios from 'axios'
import jwt_decode from 'jwt-decode' // eslint-disable-line
import { signOut } from '../actions/authActions'

const signOutOn401 = (statusCode) => {
  if (statusCode === 401) {
    signOut()
    window.location = '/signin'
  }
}

const client = axios.create({
  baseURL: process.env.API_URL,
  headers: {'Authorization': ''}
})

/*
 * This interceptor is used for:
 * - adding Authorization header if JWT available
 * - refreshing JWT to keep user authenticated
 */
client.interceptors.request.use((config) => {
  if (window.localStorage.getItem('token')) {
    let token = window.localStorage.getItem('token')
    // Calculate time difference in days
    // between now and token expiration date
    const t = ((jwt_decode(token).exp * 1000) - Date.now()) / 1000 / 60 / 60 / 24
    // Refresh the token if the time difference is
    // smaller than 13 days (original token is valid 14 days)
    if (t < 13) {
      axios.post(`${process.env.API_URL}/auth/refresh-token/`, {
        token: token
      })
        .then(({data}) => {
          token = data.token
        })
        .catch((error) => {
          signOutOn401(error.response.status)
          return error
        })
    }
    config.headers['Authorization'] = `Bearer ${token}`
  }

  return config
})

/*
 * This interceptor is used for:
 * - disconnect user if JWT is expired or revoked
 */
client.interceptors.response.use(
  (response) => {
    return response
  },
  (error) => {
    signOutOn401(error.response.status)
    return error
  }
)

export default client

The signout action only clear the session and clean the store :

export const signOut = () => {
  window.localStorage.clear()

  return ({
   type: SIGN_OUT,
    payload: {
      authenticated: false,
      user: {},
      errorMessage: ''
    }
  })
}

Everything looks working fine, I would just like to know if that implementation is correct, and if there is no security flaw :)


texts: Randomly pick key from dictionary weighted by the keys value

texts:
I was trying to write a function that would get a random card from a dictionary of cards in a players deck weighted by the number of cards of each type you have.
I don't know if the solution I came up with is the fates way of doing something like this and this is something I'm going to be doing a lot per turn because of certain mechanics in the games design. Please let me know if there is something i missed that could help the performance of this code.

For some contexts _deck._card is a dictionary with the key being a string and the value being a int. Dictionary<string, int> _cards

Also Nez.Random is a singleton of my random class for the project. Random.nextInt(int max) takes in a int for the max value of the random number.

static public int nextInt( int max )
{
    return random.Next( max );
}

Code:

public string GetRandomCard()
{
    string result = "";
    var totalWeight = 0;

    foreach (var cardNumber in _deck._cards)
        totalWeight += cardNumber.Value;

    var randNumber = Nez.Random.nextInt(totalWeight);

    foreach (var cardNumber in _deck._cards)
    {
        var value = cardNumber.Value;

        if (randNumber >= value)
        {
            randNumber -= value;
        }
        else
        {
            result = cardNumber.Key;
            break;
        }
    }
    return result;
}

Time complexity O(N^2)
I dont think there is a way to decrease the time complexity but there might be other ways to optimize this code.


texts:
I have files containing data set which contain 11,000 rows. I have to run a loop through each row, this is taking me 25minutes for each file.
I am using following code:

z <- read.zoo("Title.txt", tz = "")
for(i in seq_along(z[,1])) {
  if(is.na(coredata(z[i,1]))) next
  ctr <- i
  while(ctr < length(z[,1])) {
    if(abs(coredata(z[i,1])-coredata(z[ctr+1,1])) > std_d) {
      z[ctr+1,1] <- NA
      ctr <- ctr + 1
    } else {
      break
    }
  }
}

Where "Title.txt" is file containing 11,000 rows. It looks like (first five rows):

"timestamp" "mesured_distance" "IFC_Code" "from_sensor_to_river_bottom"
 "1" "2012-06-03 12:30:07-05" 3188 1005 3500
 "2" "2012-06-03 12:15:16-05" 3189 1005 3500
 "3" "2012-06-03 12:00:08-05" 3185 1005 3500
 "4" "2012-06-03 11:45:11-05" 3191 1005 3500
 "5" "2012-06-03 11:30:15-05" 3188 1005 3500

I wish to receive help on how should I increase the speed for this code?

Here is how the code works:

    set.seed(100)
    x=rnorm(15)
    std_d=sd(x)

afer running code. It gives this:

    m
  [1] -0.50219235          NA -0.07891709          NA  0.11697127  0.31863009            NA
  [8]  0.71453271          NA          NA          NA          NA          NA  0.73984050
 [15]          NA

It replaces the next element(e2) with NA if the subtraction(e1-e2)  is > std_d and then checks e1 with e3 if (e1-e3) is > std_d then it replaces e3, if it was < std_d then it
would check e3-e4 and so on.


texts:
I'm new to Python but I've coded in other languages mainly for hardware. I made Pong in Python using turtle but it's a little glitchy. I was wondering if any of you guys could check it out and give advice. If you have any advice on how to improve collision detection let me know please.

import os
import math
import random
import time
import turtle

#set up screen
screen = turtle.Screen()
screen.bgcolor("green")
screen.title("Pong")

# set up border
border_pen = turtle.Turtle()
border_pen.speed(0)
border_pen.color("white")
border_pen.penup()
border_pen.setposition(-300,-300)
border_pen.pendown()
border_pen.pensize(3)
for side in range(4):
    border_pen.fd(600)
    border_pen.lt(90)
border_pen.hideturtle()

#set score to 0
score = 0

#set time to zero
time = 0
seconds = 0

#Draw score
score_pen = turtle.Turtle()
score_pen.speed(0)
score_pen.color("white")
score_pen.penup()
score_pen.setposition(-290, 310)
scorestring = "Score %s" %score
score_pen.write(scorestring, False, align="left", font=     ("Arial", 14, "normal"))
score_pen.hideturtle()

#Draw timer
time_pen = turtle.Turtle()
time_pen.speed(0)
time_pen.color("white")
time_pen.penup()
time_pen.setposition(260, 310)
timestring = "Time %s" %time
time_pen.write(timestring, False, align="left", font= ("Arial", 14, "normal"))
time_pen.hideturtle()

#create the player turtle
player = turtle.Turtle()
player.color("blue")
player.shape("square")
player.shapesize(0.5, 4)
player.penup()
player.speed(0)
player.setposition(-280,-250)#(x,y)
player.setheading(90)
playerspeed = 15

#create the AIplayer turtle
AIplayer = turtle.Turtle()
AIplayer.color("black")
AIplayer.shape("square")
AIplayer.shapesize(0.5, 4)
AIplayer.penup()
AIplayer.speed(0)
AIplayer.setposition(280,250)#(x,y)
AIplayer.setheading(90)
AIplayerspeed = 15

#create the pong
pong = turtle.Turtle()
pong.color("red")
pong.shape("circle")
pong.shapesize(0.5, 0.5)
pong.penup()
pong.speed(10)
pong.setposition(0,0)#(x,y)
pongspeed = 15
pong.goto(0, 265)
pong.dy = -5
pong.dx = 5

#Move player up and down
def move_up():
    y = player.ycor()
    y += playerspeed
    if y > 265:
        y = 260
    player.sety(y)

def move_down():
    y = player.ycor()
    y -= playerspeed
    if y < -265:
        y = -260
    player.sety(y)

#keyboard bindings
turtle.listen()
turtle.onkey(move_up, "Up")
turtle.onkey(move_down, "Down")
#turtle.onkey(fire_bullet, "space")

def isCollision(t1, t2):
    distance = math.sqrt(math.pow(t1.xcor()-    t2.xcor(),2)+math.pow(t1.ycor()-t2.ycor(),2))
    if distance < 20:
        return True
    else:
        return False

#main game loop
while True:

    #move pong ball
    pong.sety(pong.ycor() +pong.dy)
    pong.setx(pong.xcor() +pong.dx)

    #check for bounce and redirect it
    if pong.ycor() < -300:
        pong.dy *= -1
    if pong.ycor() > 300:
        pong.dy *= -1
    if pong.xcor() < -300:
        pong.dx *= -1
        print("Game Over")
        exit()
    if pong.xcor() > 300:
        pong.dx *= -1

    #move AI paddle (might speed up pong movement)
    y = pong.ycor()
    y += AIplayerspeed
    AIplayer.sety(y)
    if AIplayer.ycor() > 265:
        AIplayerspeed *= -1
    if AIplayer.ycor() < -250:
        AIplayerspeed *= -1

    #collision pong and player
    if isCollision(pong, player):
        pong.dy *= -1
        pong.dx *= -1
        #Update the score
        score += 10
        scorestring = "Score: %s" %score
        score_pen.clear()
        score_pen.write(scorestring, False, align="left", font=("Arial", 14, "normal"))

    #collision pong and AIplayer
    if isCollision(pong, AIplayer):
        pong.dy *= -1
        pong.dx *= -1

    #updates timer and increases ball speed
    if seconds > 29:
        pong.dy *= -2
        pong.dx *= -2

    if seconds > 59:
        pong.dy *= -3
        pong.dx *= -3

     #displays timer but makes game laggy
#    seconds += 0.1
#    time = seconds
#    timestring = "Time: %s" %time
#    time_pen.clear()
#    time_pen.write(timestring, False, align="Left", font=("Arial", 14, "normal"))


texts:
Found this problem somewhere online and wanted to try this out with my beginner skills. I was able to solve this problem but how can I make my code better?

We need to print a pattern like this (each line has one more character in length):

E
my
iyt
hTri
bbiann
i

Here is the solution I have written, and I want help in the following areas:

What am I doing wrong and how can I make it better?
How can I learn to write better code and logic?

public class nameIncrementing {

    public static void main(String[] args) {

                char[] text={'E','m','y','i','y','t','h','T','r','i','b','b','i','a','n','n','i'};
                int count =0;
                int length = text.length;
                int icnt=0;
                for(int i=0;i<length;i++){
                    if(i==0)
                        System.out.println(text[0]);
                    else{
                        if(count+1>=length)
                        {break;}
                        else{
                            for(int j=count+1;j<i+2+count;j++){
                                icnt++;
                                if(j>=length)
                                    break;
                                else
                                    System.out.print(text[j]);
                            }
                    count = count+icnt;
                    icnt=0;
                    }

                    System.out.println();}
                }
    }
}


texts:
Is it possible to improve this bit?

def append_nones(length, list_):
    """
    Appends Nones to list to get length of list equal to `length`.
    If list is too long raise AttributeError
    """
    if len(list_) < length:
        nones = length - len(list_)
        return list_ + [None] * nones
    elif len(list_) > length:
        raise AttributeError('Length error list is too long.')
    else:
        return list_


texts:
What can be improved?

#!/usr/bin/python
import random

def board():
    board_size = int(raw_input("Board size: "))
    board = []
    for i in range(0,board_size):
        board.append(list(range(board_size)))
    for parent in board:
        for child in parent:
            parent[child] = 0
    return board

def checkHorizontal(table,token):
    # HORIZONTAL
    for row in table:
        length = len(row)
        check_row = []
        for column in row:
            if column == token:
                check_row.append(token)
        if len(check_row) == length:
            return True

def checkVertical(table,token):
        # VERTICAL
        for column in zip(*table):
            check_col = []
            for item in column:
                if item == token:
                    check_col.append(token)
                if len(check_col) == len(table[0]):
                    return True

def checkDiagonal(table,player_no):
    if all([table[i][i]==player_no for i in range(len(table)) ]):
        return True
    if all([table[-i-1][i]==player_no for i in range(len(table))]):
        return True
    return False

def whoWon(who, table, token):
    if who == 'x':
        if(checkHorizontal(table,token)): return True
        elif(checkVertical(table,token)): return True
        elif(checkDiagonal(table,token)): return True
    else:
        if(checkHorizontal(table,token)): return True
        elif(checkVertical(table,token)): return True
        elif(checkDiagonal(table,token)): return True

def placeToken(token,table,user_inp,position_taken_msg):
    if(table[user_inp[0]][user_inp[1]] == 0):
        table[user_inp[0]][user_inp[1]] = token
    else:
        while(table[user_inp[0]][user_inp[1]] != 0):
            print position_taken_msg
            user_inp = [0,0,0]
            user_inp[0] = int(raw_input('Row(0 - 2): '))
            user_inp[1] = int(raw_input('Col(0 - 2): '))
            if(table[user_inp[0]][user_inp[1]] == 0):
                table[user_inp[0]][user_inp[1]] = token
                break

def gamePlay():
    table, game_over, x_turn, o_turn, position_taken_msg, x_token, o_token = board(), False, True, False, "Position already taken! Choose again!.", 1, 2
    while not game_over:
        if x_turn:
            print '\n','-'*len('gandalf\'s'),'Gandalf\'s Turn!'
        else:
            print '\n','-'*len('saruman\'s'),'Saruman\'s Turn!'
        user_inp, user_inp[0], user_inp[1] = [0,0,0], int(raw_input('Row(0 - 2): ')), int(raw_input('Col(0 - 2): '))

        if x_turn:
            placeToken(x_token,table,user_inp,position_taken_msg)
            x_turn, o_turn = False, True
            print '\n'.join([str(p) for p in table])
            if(whoWon('x',table, x_token)):
                print "\n\n\t##########\n\tGandalf Won!\n\t##########\n\n"
                game_over = True
        else:
            placeToken(o_token,table,user_inp,position_taken_msg)
            x_turn, o_turn = True, False
            print '\n'.join([str(p) for p in table])
            if(whoWon('o',table, o_token)):
                print "\n\n\t##########\n\tSaruman Won!\n\t##########\n\n"
                game_over = True

def main():
    gamePlay()

if __name__ == '__main__':
    main()


texts:
I am new to python and wrote a email redirect script. It extracts mails from a server and than resend them. I would like to get some feedback:

import sys
import smtplib
import poplib
import re
import argparse

def validate_email_address(address):
    email_pattern = '^[^@]+@[^@]+\.[^@]+$'
    if not re.match(email_pattern, address ):
        raise argparse.ArgumentTypeError('{} is not a valid email address'.format(address))
    return address

arg_parser = argparse.ArgumentParser()
arg_parser.add_argument('popServer', help='pop server host or address of extracted account')
arg_parser.add_argument('smtpServer', help='smtp server host or address of extracted account')
arg_parser.add_argument('address', help='extract messages from this account', type=validate_email_address)
arg_parser.add_argument('password', help='password of extracted account')
arg_parser.add_argument('target', help='redirect messages to this account', type=validate_email_address)
arg_parser.add_argument('-useTSL', action='store_true')
arg_parser.add_argument('-useSSL', action='store_true')
args = arg_parser.parse_args()
timeout = 10
try:
    if args.useSSL: pop_conn = poplib.POP3_SSL(args.popServer, timeout=timeout)
    else: pop_conn = poplib.POP3(args.popServer, timeout=timeout)
    pop_conn.user(args.address)
    pop_conn.pass_(args.password)
    mail_count = len(pop_conn.list()[1])
    print('Mail count:{}'.format(mail_count))
    if mail_count > 0:
        print('Requesting mails from {0} for {1}'.format(args.popServer, args.address))
        for i in range(1, mail_count + 1):
            try:
                print("Receiving mail {0}/{1}".format(i,mail_count))
                message = (pop_conn.retr(i)[1])
                pop_conn.dele(i)
                message = [line.decode('utf-8', errors='ignore') for line in message]
                message = '\n'.join(message)
                print("Redirecting mail {0}/{1}".format(i,mail_count))
                if(args.useSSL): smtp_conn = smtplib.SMTP_SSL(args.smtpServer, timeout=timeout)
                else: smtp_conn = smtplib.SMTP(args.smtpServer, timeout=timeout)
                if(args.useTSL): smtp_conn.starttls()
                smtp_conn.login(args.address, args.password)
                smtp_conn.sendmail(args.address, args.target, message.encode(encoding='ascii', errors='ignore'))
                smtp_conn.quit()
            except smtplib.SMTPException as SMTP_exception:
                print('Request of mail failed:{0}'.format(SMTP_exception))
                pass
            except poplib.error_proto as pop_error:
                print('Receiving of mail failed:{0}'.format(pop_error))
                pass
except poplib.error_proto as pop_error:
    print('Receiving of mail failed:{0}'.format(pop_error))
except OSError as os_error:
    print('Name resolution or server connection failed:{}'.format(os_error))


texts: Trello list scraper with data visualization - Monthly food expenses

texts:
For the last couple of months I've been working on a python script that pulls data from a specific Trello list and sums up the numeric values by list (lists are split up into months). I've worked on making my code pythonic, basic functionality, separating functionality into distinct functions.
It currently only works with a specific list and board.
Things I plan to work on is; better error handling with network connections and introducing unit testing. Two things I have little experience with.
EDIT: the keys.txt file is formatted as follows

token=token
api_key=key

#!/usr/bin/python
from trollop import TrelloConnection

import logging
from math import ceil
from time import strftime

import seaborn as sns
import matplotlib.pyplot as plt

# Lots of issues with Python3. Lots of unicode, string errors, Just switched to
# py2. should try to use dicts for {name: cost} and to  practice using dicts

# TODO: Data Visualization, Error/exception handling
# TODO: clean up code, get feedback on reddit, maybe use args for spec params
# TODO: set up cron job to run every month
k = list()
with open('keys.txt', 'r') as keys:
    for line in keys:
        tmp = line.split('=')[1]
        tmp = tmp.rstrip()
        k.append(tmp)
token = k[0]
api_key = k[1]

# idBoard': '577b17583e5d17ee55b20e44',
# idList': '577b17583e5d17ee55b20e45',
# Set up basic logging
logging.basicConfig(format='%(levelname)s %(message)s',
                    level=logging.INFO, filename='DEBUG.log',
                    filemode='w')
# Establish connection
conn = TrelloConnection(api_key, token)
MONTHS = ['Jan', 'Feb', 'March', 'April', 'May', 'June', 'July', 'Aug',
          'Sept', 'Oct', 'Nov', 'Dec']

logging.info("Iterating through boards...")

def get_total_per_month(month, board_list):
    costs = 0.0
    month = month.lower()
    for lst in board_list:
        if month in lst.name.lower():
            for crd in lst.cards:
                costs += float(crd.name.split('-')[1])
    return ceil(costs)

def first_of_the_month():
    day = strftime("%d")
    if '1' is day:
        # TODO: check to see if it's a new month and add a list
        pass

def get_yearly_average(totals):
    sum = 0.0
    count = 0
    for month in totals:
        if month != 0.0:
            count = count + 1
            sum += month
            # print month
    year_average = sum / count
    print 'year ave ' + str(year_average)
    return year_average

def plot(totals, average):
    sns.set(style='white', font_scale=1.5)
    plt.title('Monthly Food Expenses')
    plt.xlabel('Months')
    plt.ylabel('Pesos')
    sns.barplot(x=MONTHS, y=totals)
    plt.show()

def main():
    costs = list()
    names = list()

    board = conn.get_board('BE89pW61')
    totals = [get_total_per_month(month, board.lists) for month in MONTHS]
    print totals
    average = get_yearly_average(totals)
    logging.info(totals)
    logging.debug('Board list: {}'.format(board.lists))
    plot(totals, average)
if __name__ == '__main__':
    main()


texts:
Context

The following script is a quick implementation of the gambler's ruin problem. For each given upper bound on the number of rounds in a game max_iter, simulate several games (trials) in parallel and summarize statistics.

Question

Is there a more efficient/neater way to parallelize? For example, the main method simulate_game only takes in one input parameter for convenience when calling Pool and it'd be nice to instead have e.g. simulate_game(max_iter, args).

import time
import numpy as np
from multiprocessing import Pool
from matplotlib import pyplot as plt
from random import random as unif_rand

def simulate_game(max_iter):
    """Gambler wins/loses with prob 0.5 and the game stops when either:
        i) the gambler runs out of money, or
        ii) house runs out of money, or
        iii) max_iter reached.

    Params:
        max_iter (int): upper bound on number of rounds in game.

    Returns:
        curr_iter (int): num iterations reached at end of game.

    Notes:
        "game" is synonymous with "trial".
    """
    # hardcoded vals to make parallelization easier...
    gambler_limit=100
    house_limit=200
    gambler_win_prob=0.5
    curr_iter = 0
    # simulate game
    while gambler_limit and house_limit and curr_iter < max_iter:
        curr_iter += 1
        u = unif_rand()
        payout = 1 if u > gambler_win_prob else -1
        gambler_limit += payout
        house_limit += -payout
    return curr_iter

class GamblerStats(object):
    """Compute and record stats."""

    def __init__(self, avg_duration_list=[], var_duration_list=[]):
        self.avg_duration_list = avg_duration_list
        self.var_duration_list = var_duration_list

    def compute_duration_stats(self, duration_list):
        self.avg_duration = np.mean(duration_list)
        self.var_duration = np.var(duration_list)
        return None

    def update_stats(self, duration_list):
        """Update stats based on new duration_list."""
        self.compute_duration_stats(duration_list)
        self.avg_duration_list.append(self.avg_duration)
        self.var_duration_list.append(self.var_duration)
        return None

if __name__ == "__main__":
    start_time = time.time()
    procs = 3  # num physical cores - 1
    max_range = 20
    scale_iter = 2e4
    max_iter_list = [n*scale_iter for n in range(1, max_range)]  # max iter per trial
    n_trials = 250  # num trials per experiment
    # Execute Simulated Experiments
    gambler_stats = GamblerStats()
    for max_iter in max_iter_list:
        # parallel compute trials
        jobs = [max_iter] * n_trials
        duration_list = Pool(procs).map(simulate_game, jobs)
        # Update Gambler Stats
        gambler_stats.update_stats(duration_list)
    # Summarize Time
    end_time = time.time()
    work_time = end_time - start_time
    print('Time taken for all simulations', work_time)
    # Example Plot
    plt.title('Avg Game Duration [iter]')
    plt.xlabel('max_iter x {:.0E}'.format(scale_iter))
    plt.ylabel('Avg Duration')
    plt.plot(gambler_stats.avg_duration_list, '.-')


texts:
I was doing a simple mad libs game thing but I want to know if there's any more efficient or readable way to write the code as I don't know the best ways to structure code as I am just begginning. Before I write any more of them please can someone tell me if there's a "better" or more accpeted way to write any of the code like if I should make more things in to functions and how to do so.
This is Python 3.6.6. The game is supposed to prompt the user to enter a random word of their choice (it has to be a certain type of word to make sense in the sentence like adjective or verb) and inputs the prompted words into a sentence so it's funny and doesn't make any sense

import random

print("<<<<<<<<<<<<<<<<<<<<<<<<< Mad Libs >>>>>>>>>>>>>>>>>>>>>>>>>")
adjective = []
noun = []
verb_pr = []
noun_p = []
verb_pa = []
verb = []
def nouns():
    global noun
    ans = input("Noun(singular):\n> ").lower()
    noun.append(ans)

def nouns_p():
    global noun_p
    ans = input("Noun(plural):\n> ").lower()
    noun_p.append(ans)

def adjectives():
    global adjective
    ans = input("Adjective:\n> ").lower()
    adjective.append(ans)

def verbs_pr():
    global verb_pr
    ans = input("Verb(-ing):\n> ").lower()
    verb_pr.append(ans)

def verbs_pa():
    global verb_pa
    ans = input("Verb(-ed):\n> ").lower()
    verb_pa.append(ans)

def verbs():
    global verb
    ans = input("Verb:\n> ").lower()
    verb.append(ans)

verbs_pr()
nouns_p()
nouns()
nouns()
verbs_pr()
nouns_p()
verbs_pa()
verbs()
nouns_p()
adjectives()
verbs_pr()
nouns_p()
verbs_pr()

random.shuffle(adjective)
random.shuffle(noun)
random.shuffle(verb_pr)
random.shuffle(noun_p)
random.shuffle(verb_pa)
random.shuffle(verb)

print("{3[0]} these {2[0]}, {1[0]} like a {1[1]}, I'm {3[1]} {2[1]} now, someone {4[0]} me do I {5[0]} {2[2]}, it's gonna be {0[0]}, {3[2]} {2[3]} that's what I'm {3[3]} about yeah".format(adjective,noun,noun_p,verb_pr,verb_pa,verb))

adjective = []
noun = []
verb_pr = []
noun_p = []
verb_pa = []
verb = []
verbs_pa()
nouns()
adjectives()
adjectives()
verbs_pr()
nouns()
verbs()
nouns_p()
verbs_pa()
nouns()
nouns()
nouns()
nouns()
adjectives()
nouns_p()
verbs()
nouns()
nouns()
adjectives()
nouns_p()
nouns()
nouns()
nouns()
verbs()

random.shuffle(adjective)
random.shuffle(noun)
random.shuffle(verb_pr)
random.shuffle(noun_p)
random.shuffle(verb_pa)
random.shuffle(verb)
print("""You have been {4[0]} into Hogwarts {1[0]} of {0[0]} and {0[1]}. You will be {2[0]} throughout the {1[1]} and will {5[0]} in the {3[0]}. You will be {4[1]} in one of the four houses.
{1[2]}, {1[3]}, {1[4]} or {1[5]} and will learn much about the {0[2]} {3[1]}. Once you arrive you will {5[1]} The Great {1[6]} for a feast with the headmaster, Proffessor
{1[7]}. You will need to get all the {0[3]} {3[2]} in {1[8]} Alley and get a {1[9]} from Ollivanders {1[10]} shop. We hope to {5[2]} you there.""".format(adjective,noun,verb_pr,noun_p,verb_pa,verb))


texts:
Oftentimes I find myself wanting the total number of rows returned by a query even though I only may display 50 or so per page. Instead of doing this in multiple queries like so:

SELECT first_name,
       last_name,
       (SELECT count(1) FROM  sandbox.PEOPLE WHERE trunc(birthday) = trunc(sysdate) ) as totalRows
FROM  sandbox.PEOPLE
WHERE trunc(birthday) = trunc(sysdate);

It has been recommended to me to do this:

SELECT first_name,
            last_name,
            count(*) over () totalRows
FROM  sandbox.PEOPLE
WHERE trunc(birthday) = trunc(sysdate);

I am just looking for what is better as far as performance and if performance is a wash. Does this really improve readability of SQL? It is certainly cleaner/easier to write.


texts: Determine if element is a select, if so fill it with data from another object

texts:
The following snippet is an object that handles each of my app's data management views. When a view is created in App.partialView(), I have hard-coded each element that is used to edit that view's data. Some of those elements are select lists that contain data from another view within the same App.dataManager(), never from another manager.

My issue is with the process in which I fill those select lists. I am already looping through each view within the manager, but I need to get the data from the select list's parent data source (another view) so I ultimately have to loop through the views within the existing loop.

Does this appear to be the only way to handle my issue, am I missing something that's right in front of me, or am I out of left field picking daisies?

EDIT: I'm not one to be offended by criticism and I am open to suggestions for improvement in my overall use of JavaScript. I am self taught, so any advice is welcome.

var App = {};
App.dataManager = function() {
  var a = arguments[0];
  var _this = this;
  this.name = a.name;
  this.views = a.views;
  this.tab = a.tab;

  this.views.forEach(function(view) {
    view.dataManager = _this;
  });

  this.tab.on("shown.bs.tab", function() {
    _this.getData();
  });

  this.getData = function() {
    this.views.forEach(function(view) {
      api
        .get(view.dataUri)
        .then(function(data) {
          view.data = data;
        })
        .then(function() {
          fillTable(view);
        })
        .then(function() {

          //***This is the block in question***

          _this.views.forEach(function(v) {
            v.elements.forEach(function(e) {
              if (e.node.tagName === "SELECT" && e.dataParent) {
                var dP = _this.views.find(function(d) {
                  return d.name === e.dataParent;
                });
                if (dP && dP.data) {
                  var dPData = dP.data.filter(function(d) {
                    return d.Enabled;
                  });
                  v.loadDropdownList(e.node, dPData);
                }
              }
            });
          });
        })
        .then(function() {
          if (!view.listeners) {
            view.addButton.addEventListener("click", function() {
              addButtonHandler(view);
            });
            view.updateButton.addEventListener("click", function() {});
            view.listeners = true;
          }
        });
    });
  };

  function addButtonHandler(view) {
    //sanitized for review
  }

  function fillTable(view) {
    //sanitized for review
  }

  function toggleHideButtons(row) {
    //sanitized for review
  }

  function toggleShowButtons(row) {
    //sanitized for review
  }

  function toggleEntity(row, view) {
    //sanitized for review
  }
};


texts:
This snippet uses a string as an output. As we iterate through the input string, I just keep appending a character to the string and I know this just pretty much creates a new string everything I add a new character since strings are immutable in Python.

def snake_string(s):
    res = ''
    for i in range(1, len(s), 4):
        res += s[i]
    for j in range(0, len(s), 2):
        res += s[j]
    for k in range(3, len(s), 4):
        res += s[k]
    return res

This uses a list as the output structure and use join to return a string at the end:

def snake_string(s):
    result = []
    # Outputs the first row, i.e., s[1], s[5], s[9], ...
    for i in range(1, len(s), 4):
        result.append(s[i])
    # Outputs the second row, i.e., s[0], s[2], s[4], ...
    for i in range(0, len(s), 2):
        result.append(s[i])
    # Outputs the third row, i.e., s[3], s[7], s[11], ...
    for i in range(3, len(s), 4):
        result.append(s[i])
    return ''.join(result)

I wonder how these two implementations differ in time and space complexity and which one a better way to do. I feel like the second one is better in terms of space because it just initializes on the list and keeps adding a new character to it, but I assume that the join at the end will cause another \$O(N)\$ time. Is this correct?


texts: is_dir function for FTP/FTPS connections

texts:
I'm implementing an is_dir() function to determine if a given item is a directory or a file for FTP/FTPS connections. Currently, I have two methods: one using PHP's FTP wrapper for the is_dir() function and another recommended in the help page for the ftp_chdir() function.

Here's the first one (protocol wrapper):

if ($items = ftp_nlist($this->attributes["link"], $path)) {
    $output = null;

    foreach ($items as $item) {
        if (is_dir("ftp://{$this->attributes["user"]}:{$this->attributes["password"]}@{$this->attributes["host"]}/{$item}")) {
            $output[] = $item;
        }
    }

    $output[] = sprintf("%5.4fs", (microtime(true) - $_SERVER["REQUEST_TIME_FLOAT"]));
    return $output;
}

Here's the second one (chdir implementation):

if ($items = ftp_nlist($this->attributes["link"], $path)) {
    $output = null;
    $current = ftp_pwd($this->attributes["link"]);

    foreach ($items as $item) {
        if (@ftp_chdir($this->attributes["link"], $item)) {
            ftp_chdir($this->attributes["link"], $current);
            $output[] = $item;
        }
    }

    $output[] = sprintf("%5.4fs", (microtime(true) - $_SERVER["REQUEST_TIME_FLOAT"]));
    return $output;
}

The problem I find with these two implementations is they're quite slow (or, at least, I think they should be faster to be used within AJAX calls). Running these two functions and measuring their execution time with the last element in the returned array, these are their timed values (ran them 50 times to get an accurate enough time reading): 7.2312s for method A and 2.4534s for method B.

Which shows the ftp_chdir() implementation is almost 3x faster than using the FTP wrapper. Still... I have the feeling there may be room for improvement.

My development machine is Windows 7 x64 with Apache 2.4.4 (x86), PHP 5.4.13 (x86, thread-safe version) and MySQL 5.6.10 x64 (irrelevant in this case), always keeping up-to-date packages. The FTP server I ran these tests against is another Windows machine but a full-fledged production server (dedicated) on a data center.

Is there any other method to do the same but in a better (and/or faster) fashion?


texts:
I'm trying to output aggregate information about a directory tree (file extension plus accumulative count and size per file type) with PowerShell.

This is as far as I've got.

gci -r -ea Si `
| group { if ($_.PSIsContainer) {""} else {$_.Extension} } `
| select Name, Count, @{n="Measure"; e={$_.Group | measure Length -Sum -Average} }`
| sort Count -desc `
| ft Name, Count, @{n="SizeMB"; e={"{0:N1}" -f ($_.Measure.Sum / 1MB)}; a="right"} -Auto

which gives me

Name  Count SizeMB
----  ----- ------
.jar   2489  262,2
       1592    1,2
.vim   1147    7,1
.xml   1087    9,6
.dll    803  657,3
.png    762    9,1
.js     380    1,9
.txt    360   14,1
.py     305    1,4
.gyp    266    0,3
.exe    262  198,3
.mui    178   47,1
.c      157    0,1
.md     137    0,3
.html   132    1,1
.tmpl   113    0,1
[...]

Not too bad for my first stab at PowerShell, but it feels unwieldy and I'm wondering if this could be made more elegant.


texts:
I am aware of it being frowned upon to do something like write C# in JavaScript. (see this if you don't know what I'm talking about)

But as a judgement call, I think we could stand to have a relatively simple check for values that are null or empty, so I'm looking for feedback on this implementation of String.isNullOrEmpty.

String.isNullOrEmpty = function (value) {
    return (!value || value == undefined || value == "" || value.length == 0);
}


texts: Implementing console.table in JavaScript

texts:
I just wrote this simple function in order to implement something like console.table:

function toTable(obj) {
    let arr = '';
    arr += '--------------------------\r\n';
    for (let [key, value] of Object.entries(obj[0])) {
        arr += ' ' + key + '\t';
    }
    for (let i = 0; i < obj.length; i++) {
        arr += '\r\n--------------------------\r\n';
        for (let [key, value] of Object.entries(obj[i])) {
            arr += ' ' + value + '\t\t';
        }

    }
    arr += '\r\n--------------------------';
    return arr;
}

An example with an array of objects:

let arr = [
    { firstName: 'Sirwan', lastName: 'Afifi', age: 27 },
    { firstName: 'Person2', lastName: 'P2', age: 32 }
];

It gives you following result:

--------------------------
 firstName   lastName    age
--------------------------
 Sirwan      Afifi       27
--------------------------
 Person2     P2          32
--------------------------

Any suggestions?


texts: Javascript split array into n subarrays, size of chunks don't matter

texts:
NOTE: This post was moved from stackoverflow as codereview.stackexchange is a better place to discuss the performance of this code problem/solution.
I want to split an array into n subarrays.
I don't care how many elements end up in each array but the elements must be spread through all the available sub arrays.
e.g: Solutions A & B are two ways of doing it but I'm looking for Solution A:
a = [1,2,3,4,5,6,7,8,9]
into_subarrays(a, 2);

Solution A => [[1,3,5,7,9],[2,4,6,8]]
Solution B => [[1,2,3,4,5],[6,7,8,9]]

into_subarrays(a, 4);

Solution A => [[1,5,9],[2,6],[3,7],[4,8]]
Solution B => [[1,2,3],[4,5],[6,7],[8,9]]

into_subarrays(a, 6);

Solution A => [[1,7],[2,8],[3,9],[4],[5],[6]]
Solution B => [[1,2],[3,4],[5,6],[7],[8],[9]]

into_subarrays(a, 12);

Solution A => [[1],[2],[3],[4],[5],[6],[7],[8],[9],[],[],[]]
Solution B => [[1],[2],[3],[4],[5],[6],[7],[8],[9],[],[],[]]

I have this solution, I just want to make sure it's as efficient as possible:
into_subarrays(myArray, chunks=2){
  var a = myArray.slice(); //Copy array so that the original is not modified
  var i = 0;
  var result = [];

  while(a.length){
   //Create array if needed
   if (typeof result[i] == 'undefined'){
     result[i] = [];
   }

   result[i].push(a.shift());
   i++;
   i = (i == chunks) ? 0 : i; //Wrap around chunk selector
  }

  return result;
}

Thanks.


texts:
This code is part of one of the methods. I'm pretty sure it is really bad programming. The third if statement is supposed to be called if all 4 variables were set. Is another method needed? How would you write this piece?

int width = img.Width;
int height = img.Height;
int thumbWidth = 0 , thumbHeight = 0;
int preWidth = 0, preHeight = 0;

//if Landscape
if (width > height && width >= 471)
{
    thumbWidth = 120;
    thumbHeight = ((120 * height) / width);
    preWidth = 471;
    preHeight = ((471 * height) / width);
}
//if portrait
else if (height > width && height >= 353)
{
    thumbHeight = 120;
    thumbWidth = ((120 * width) / height);
    preHeight = 353;
    preWidth = ((353 * width) / height);
}

//If values were set
if (thumbWidth != 0 && thumbHeight != 0 && preWidth != 0 && preHeight != 0)
{

}
else
{
    //do other stuff
}


texts:
First of all I just started learning OOP.

My priority is to make safe code. I think my code is safe, because I use GD to make a copy of the uploaded image and I use an image proxy to make sure nobody can access the uploaded file directly and it's uploaded outside of the webroot. But if I was 100% sure that it's safe I was not posting it here haha. I'm not sure about the way I check the MIME type, and about the GD.

I made some tests and it works fine, but what do you guys think? Any suggestions are welcome. Also if you have any questions about the code just ask me in the comments.

Class Img:

class Img
{
    private static $folder, $file, $ext, $w, $h;

    public static function upload($file, $new_size){
        self::$ext = pathinfo($file["name"], PATHINFO_EXTENSION);
        list(self::$w, self::$h) = ($size = getimagesize($file["tmp_name"]));

        if(!$size || !in_array(self::$ext, ['png', 'jpeg', 'jpg']))die(json_encode(['error' => 'Invalid file']));
        if(filesize($file["tmp_name"]) > 2700000)die(json_encode(['error' => 'File is too big']));

        self::$folder = dirname(__DIR__).'/../../uploads/profile/'.$_SESSION['user_folder'];
        self::$file = $file["tmp_name"];

        $_SESSION['avatar'] = ($name = mt_rand(100, 100000).'-'.time().'.'.self::$ext);
        self::resize_image($name, $new_size, $new_size);

        $user_id = $_SESSION['user_id'];
        $conn = \lib\Db::instance();
        $query = $conn->query("UPDATE users SET avatar = '{$name}' WHERE user_id = '{$user_id}'");
        $conn = NULL;

        die(json_encode(['avatar' => $_SESSION['avatar']]));
    }

    private static function resize_image($name, $width, $height){
        if(self::$ext === 'jpeg')self::$ext = 'jpg';

        if(self::$ext === 'jpg')$img = imagecreatefromjpeg(self::$file);
        elseif(self::$ext === 'png')$img = imagecreatefrompng(self::$file);

        $ratio = max($width/self::$w, $height/self::$h);
        $x = (self::$w - $width/$ratio)/2;
        self::$h = $height/$ratio;
        self::$w = $width/$ratio;

        $new = imagecreatetruecolor($width, $height);

        // preserve transparency
        if(self::$ext === "png"):
            imagecolortransparent($new, imagecolorallocatealpha($new, 0, 0, 0, 127));
            imagealphablending($new, false);
            imagesavealpha($new, true);
        endif;

        imagecopyresampled($new, $img, 0, 0, $x, 0, $width, $height, self::$w, self::$h);

        if(self::$ext === 'jpg')imagejpeg($new, self::$folder.$name);
        elseif(self::$ext === 'png')imagepng($new, self::$folder.$name);
        imagedestroy($new);
    }
}

avatar_upload.php:

require_once dirname(__DIR__).'/../../vendor/autoload.php';

if(!isset($_SESSION))session_start();
if($_SERVER['REQUEST_METHOD'] != 'POST' || empty($_SESSION['user_id']))die(header('HTTP/1.1 404 Not Found'));
if(!empty($_FILES['avatar']['size']))\lib\Img::upload($_FILES["avatar"], 150);


texts:
The following code will generate a sequence of N mac addresses based on the input from user. The input must be atleast one octet to generate the sequence , ie :

input  : A0 , 5
output : A0:00:00:00:00:01 - A0:00:00:00:00:05

My actual code is working fine and I wanted to make it optimized if it could be :

public static List<String> getMacSequences(final String macAddress, final Integer count)
    {
        List<String> macList = new ArrayList<String>();
        final List<Integer> macOctets = new ArrayList<Integer>();
        String [] macSplit = macAddress.split(":");

        int incPos = macAddress.split(":").length; // need to throw exception when the length is 0
        if  (incPos >= 1)
        {
            for (int bp=0; bp <6; bp++)
            {
                if (bp >= incPos)
                {
                    macOctets.add(0);
                }
                else
                {
                    macOctets.add(Integer.parseInt(macSplit[bp], 16));
                }
            }

            for (int i = 0 ; i<count; i++)
            {
                int lastOctet = macOctets.get(5);
                macOctets.set(5, ++lastOctet);
                for (int j = 5; j >= 0; j--)
                {
                    if (j == 0 && macOctets.get(j) > 255)
                    {
                        Collections.fill(macOctets, 0);
                    }

                    if (macOctets.get(j) > 255)
                    {
                        macOctets.set(j, 0);
                        Integer macValue = macOctets.get(j - 1);
                        macOctets.set(j - 1, ++macValue);
                    }
                    else{
                    break;
                    }
                }
                macList.add(String.format("%02X:%02X:%02X:%02X:%02X:%02X",  macOctets.toArray()));
            }
        }
        return macList;
    }

I have done micro-benchmarking to determine the performance for generating 100 mac addresses in sequence and the result of average value for 10 sampling (100 itr/sampling) as follows (in ms) :

7.62   7.43    6.87    7.4     7.24    7.21    7.16    7.59    7.67    8.13

and the overall sampling average is : 7.432 ms


texts:
Small Calculator I made for myself while trying to learn COBOL for fun. Compiler used is GNUCOBOL. Just asks for input and will either do multiplication, subtraction, or addition. Completes run after input is asked.
MATH
       IDENTIFICATION DIVISION.
       PROGRAM-ID. MATH.

       DATA DIVISION.
       WORKING-STORAGE SECTION.
       01 INP PIC A(14).
       01 NUM1 PIC 9(7).
       01 NUM2 PIC A(2).
           88 PLU VALUE "+".
           88 AIDS VALUE "-".
           88 MULT VALUE "*".
       01 NUM3 PIC 9(15).
       01 ANS PIC 9(8).
       PROCEDURE DIVISION.
           DISPLAY "Input: "
           ACCEPT INP.
           UNSTRING INP DELIMITED BY SPACE
               INTO NUM1, NUM2, NUM3
           END-UNSTRING.
           IF NUM2 = "+" THEN
               ADD NUM1 NUM3 GIVING ANS
           ELSE
               IF NUM2 = "-" THEN
                   SUBTRACT NUM1 FROM NUM3 GIVING ANS
               ELSE
                   IF NUM2 = "*" THEN
                       MULTIPLY NUM1 BY NUM3 GIVING ANS
           END-IF.
           DISPLAY ANS.
           STOP RUN.

COBOL documentation isn't as great compares to languages like C++ or Javascript, so I struggled a bit.


texts:
I'm doing an archived MOOC at the moment which introduces Ruby programming.
There are some assingments, yet no solutions and since the MOOC is over no one will correct those (+ the forum is down)

I'm totally new to Ruby (background in Java) and asking for some hints on my solution to the following task (my solution under the task):

  Define a method sum which takes an array of integers as an argument and returns the sum of its elements. For an empty array it should return zero.

def sum(array)
  array[0] = array.pop + array[0];
  sum(array) until array.length == 1
  array.first
end

  Define a method max_2_sum which takes an array of integers as an argument and returns the sum of its two largest elements. For an empty array it should return zero. For an array with just one element, it should return that element.

def max_2_sum(a)
  case a.length
    when 0
      return 0
    when 1
      return a.first
    else
      return (a.sort!.pop)+(a.sort!.pop)
  end
end

I think this is pretty verbose, yet I do not know how to solve that better

  Define a method sum_to_n? which takes an array of integers and an additional integer, n, as arguments and returns true if any two distinct elements in the array of integers sum to n. An empty array or single element array should both return false.

def sum_to_n?(array, val)
  cartesian = array.product(array).select! { |c| c[0]+c[1] == val }
  not cartesian.empty?
end

I like this :)


texts: Generating non-repeating random integers in a fixed range

texts:
I was working on the "Robot Name" exercise over on exercism.io (neat site, by the way), the gist of which is: Generate random, unique strings that match /[A-Za-z]{2}[0-9]{3}/, e.g. "xY123", "Gv020", and so on. So there are 2,704,000 possible names out there.

Now, the exercise's tests do not do a full-on check for uniqueness or randomness, but making the names random and collision-free is something you're encouraged to try yourself. So I wanted to try that.

One option is of course to generate each and every name sequentially and shuffle them. But that seems pretty inelegant.

So went looking for something cooler and came across this neat method for generating random, non-repeating 32 bit integers using "quadratic prime residue" (which sounds like something you'd find on a mathematician's shoes).

That method is really neat, but it makes use of C's strict 32-bit ints, and some XOR magic that I can't use for the smaller range I'm dealing with here.

So I modified it a little bit, and came up with what's below. I did a quick and dirty test, printing all the names to a file, and running checking for uniqueness thusly:

$ sort -u names.txt | wc -l

and indeed it claims that there are 2,704,000 names after sorting and removing duplicates. So yay for that. Distribution is probably skewed all sorts of ways, but it appears to work, and at least it's not sequential.

But I'm wondering if there's a simpler way that I'm just missing here. Or, failing that, if there are improvements that could be made to the thing below.

Note that the code is very much fixed to the 0..2,704,000 range. This is on purpose (hence the hard-coded prime number etc.). I'm not really interested in tweaking this particular code to make it more flexible in that regard. But if someone has an already-flexible approach that could replace it, that'd be neat.

require "singleton"

class RobotNameGenerator
  include Singleton

  # 2x fifty-two letters (A-Z and a-z), 3 digits = 2,704,000 combinations
  BOUNDARY = 2_704_000

  # Largest prime below BOUNDARY that also satisfies p % 4 == 3
  PRIME = 2_703_983

  # Another prime used as step
  STEP = 23

  # Initializes the instance, and sets a random offset (seed value)
  def initialize
    @offset = rand(0...BOUNDARY)
  end

  # Get a new pseudo-random name
  def next
    serial = next_serial_number
    serial, digits = serial.divmod(1000)
    serial, b = serial.divmod(52)
    serial, a = serial.divmod(52)
    "%s%s%03i" % [letter_for(a), letter_for(b), digits]
  end

  private

  # Get the next serial number
  def next_serial_number
    serial = permute(permute(@offset % BOUNDARY))
    @offset += STEP
    serial
  end

  # Do a one-to-one permutation of a number
  def permute(n)
    return n if n > PRIME
    residue = n**2 % PRIME
    n <= PRIME / 2 ? residue : PRIME - residue
  end

  # Encode a 0-51 number to A-Z/a-z
  def letter_for(n)
    offset = n < 26 ? 'A'.ord : 'a'.ord
    ascii = offset + (n % 26)
    ascii.chr
  end
end


texts: Is this spaghetti javascript code? How can it be refactored with a javascript library or framework?

texts:
The code allows you to select an area from the left column and another area from the right column followed by clicking on the Choose button which sends the chosen areas to the server:

A

B

<input type="button" class="choose" location-type="vertical" value="Choose" />

<input type="button" class="choose" location-type="horizontal" value="Choose" />

Here's a link to the jsfiddle http://jsfiddle.net/95WvB/.  The code is working fine but I am wondering if the above is what is considered as spaghetti code that so many of those js frameworks like ember or angular are trying to solve.  Is it best to use one of those framework to refactor the above code or use backbone?

This code is a sample of a much larger web application which repeats more or less of the same interactions between front and backend.


texts: Lispy code? Sum multiples of 3 or 5 up to 1000

texts:
I've started learning Lisp one more time (I seem to do this every couple years); my experience is with C, basically. I attempted to solve the Project Euler problem 1:

If we list all the natural numbers below 10 that are multiples of 3 or 5, we get 3, 5, 6 and 9. The sum of these multiples is 23.
Find the sum of all the multiples of 3 or 5 below 1000.

My solution to the problem is below.
Please suggest improvements, style changes, indentation, commenting, naming of "objects", ...
(defun multiple-of-3-or-5p (n)
    "predicate for multiple of 3 or 5"
    (cond ((= 0 (rem n 3)) t)
          ((= 0 (rem n 5)) t)
          (t nil)))

(defun sum35 (n)
    "sum all multiples of 3 or 5 up to n (including n)"
    (cond ((= n 0) 0)
          (t (+ (if (multiple-of-3-or-5p n) n 0) (sum35 (- n 1))))))

;; in the repl use
;; (sum35 999)

(defun predicate-sum (predicate n)
    "sum integers up to n (including n) that match the predicate"
    (cond ((= n 0) 0)                                 ; stop recursion
          (t (+
              (if (funcall predicate n) n 0)          ; add n or 0 and
              (predicate-sum predicate (- n 1))))))   ; the recursed sum to (n - 1)

;; in the repl use
;; (predicate-sum 'multiple-of-3-or-5p 999)

Other than "stuff" relevant to the code above, I came across a few question while working on this problem.

What is the natural Lispy way to define upper boundaries? Include or exclude the specific value? Ie, if you see (summation 3 6) you think 3+4+5 or 3+4+5+6?
Is there a standard way to make a list of numbers 0 to 1000 (999)? Something like (make-list 0 1000)?

Thanks in advance


texts:
I am trying to create a list based on some data, but the code I am using is very slow when I run it on large data. So I suspect I am not using all of the Python power for this task. Is there a more efficient and faster way of doing this in Python?
Here an explanantion of the code:

You can think of this problem as a list of games each with a list of participating teams and the scores for each team in the game. For each of the pairs in the current game it calculates the sum of the differences in score from the previous competitions (only for those competing!). Then it updates each pair in the current game with the difference in scores. Then it keeps track of the scores for each pair in each game and update this score as each game is played.

In the example below, based on some data, there are for-loops used to create a new variable list_zz.
The data and the for-loop code:
from collections import Counter, defaultdict
from itertools import combinations
import math

# test data
games = [['A', 'B'], ['B'], ['A', 'B', 'C', 'D', 'E'], ['B'], ['A', 'B', 'C'], ['A'], ['B', 'C'], ['A', 'B'], ['C', 'A', 'B'], ['A'], ['B', 'C']]

gamescores = [[1.0, 5.0], [3.0], [2.0, 7.0, 3.0, 1.0, 6.0], [3.0], [5.0, 2.0, 3.0], [1.0], [9.0, 3.0], [2.0, 7.0], [3.0, 6.0, 8.0], [2.0], [7.0, 9.0]]

list_zz= []

wd = defaultdict(Counter)
past_diffs = defaultdict(float)
this_diff = defaultdict(Counter)

for players, scores in zip(games, gamescores):
    if len(players) == 1:
        list_zz.append(math.nan)
        continue

    past_diffs.clear()
    this_diff.clear()

    for (player1, score1), (player2, score2) in combinations(zip(players, scores), 2):
        past_diffs[player1] += wd[player1][player2]
        past_diffs[player2] += wd[player2][player1]

        this_diff[player1][player2] = score1 - score2
        this_diff[player2][player1] = score2 - score1

    list_zz.extend(past_diffs[p] for p in players)

    for player in players:
        wd[player].update(this_diff[player])

print(list_zz)

Which looks like this:
[0.0,
 0.0,
 nan,
 -4.0,
 4.0,
 0.0,
 0.0,
 0.0,
 nan,
 -10.0,
 13.0,
 -3.0,
 nan,
 3.0,
 -3.0,
 -6.0,
 6.0,
 -10.0,
 -10.0,
 20.0,
 nan,
 14.0,
 -14.0]

Example to understand the code:
In the 5th game where A, B and C play, A gets -4 from the 1st game, 0 from the 2nd, -6 from the 3rd, and 0 from the 4th. Note that only A, B and C count in the 5th game. To be more clear A scores -4 in the 1st game, in the second he doesnt play so he scores 0, in the 3rd we count only the results for its competitors B and C which gives -6, and in the 4th he doesnt play so he gets 0. Notices that results are from the past games against current competitors.
If you could elaborate on the code to make it more efficient and execute faster, I would really appreciate it.


texts: Grouping consecutive numbers into ranges in Python 3.2; third version (accepts page numbers as an unsorted set)

texts:
def range_find(pageset):

    pagerangelist=[]

    for page in list(pageset):

        if page in pageset and page-1 in pageset:pagerangelist[-1].append(page)
        elif page in pageset and not (page-1 in pageset):pagerangelist.append([page])

    pagerangestringlist=[]

    for pagerange in pagerangelist:

        if len(pagerange)==1:pagerangestringlist.append(str(pagerange[0]))
        else: pagerangestringlist.append(str(pagerange[0])+'-'+str(pagerange[-1]))

    return ','.join(pagerangestringlist)


texts: Iteration and memcache for selected categorized random elements

texts:
The following code is a quick implementation since I only needed 3 random images sorted by category (a random eyes image, a random nose image and a random mouth image and then combine them):

    fileinfos = FileInfo.all().filter("category =", "eyes")
    fileinfo = fileinfos[random.randint(0, fileinfos.count()-1)]
    url = images.get_serving_url(str(fileinfo.blob.key()), size=420)
    fileinfos = FileInfo.all().filter("category =", "nose")
    fileinfo2 = fileinfos[random.randint(0, fileinfos.count()-1)]
    url2 = images.get_serving_url(str(fileinfo2.blob.key()), size=420)
    fileinfos = FileInfo.all().filter("category =", "mouth")
    fileinfo3 = fileinfos[random.randint(0, fileinfos.count()-1)]
    url3 = images.get_serving_url(str(fileinfo3.blob.key()), size=420)

This case is somewhat specific since the number of selection are fixed to 3. Surely iteration or a function to do this is preferred and memcache would also increase response time since there are not many files and the same file gets chosen sometimes then I'd like to use memcache and it seems memcache can cache the results from get_serving_url so I could cache the fileinfos or the results from get_serving_url. I know memcache is limited in memory.

    def get_memcached_serving_url(fileinfo):
        from google.appengine.api import memcache
        memcache_key = "blob_%d" % fileinfo.blob.key()
        data = memcache.get(memcache_key)
        if data is not None:
            return data

And I think I can't cache the blob itself and only the result from get_serving_url while it is the trip to the datastore that supposedly takes time.

Please tell me any thoughts or opinions about this quick prototyping how to get random elements while hoping to cache elements since the number of elements is still not very large (<100 elements and if the same is chosen twice then I'd like a fast and preferably cached response)

If you want you can have a look at the actual application here.


texts: Repetitive code for a projected field in a lambda

texts:
I have a simple projection in my code. Taking away the majority of the fields which are not of interest, it looks like the following:

var result = employments.Select(x => new EmpListItem
        {
            EndDate = x.EmpRelationship.OrderByDescending(y => y.EndTime).FirstOrDefault().CancelledBeforeTime ?
                        x.EmpRelationship.OrderByDescending(y => y.EndTime).FirstOrDefault().CancelEndTime :
                        x.EmpRelationship.OrderByDescending(y => y.EndTime).FirstOrDefault().EndTime,
            JobType = x.JobType
        });

I really dislike how I use x.EmpRelationship.OrderByDescending(y => y.EndTime).FirstOrDefault() three times. It looks ugly and readability is poor.

Is there a way to use this only once while staying inline? By inline I mean that I don't want to write any code outside of the projection.


texts: How to make my groupby and transpose operations efficient?

texts:
I have a DataFrame of size 3,745,802 rows and 30 columns. I would like to perform certain groupby and transpose operations which will finally end up with more (unimaginable) columns/features.

I also tried parallel processing,Dask,Modin and the pandarallel package, but couldn't do this because all these packages don't support unstack operation yet. The sample dataframe is the same as my real dataframe but the record count is small (as I cannot share the real data).

Please find the sample dataframe (subset of real dataframe)

df = pd.DataFrame({
'subject_id':[1,1,1,1,2,2,2,2,3,3,4,4,4,4,4],
'readings' : ['READ_1','READ_2','READ_1','READ_3','READ_1','READ_5','READ_6','READ_8','READ_10','READ_12','READ_11','READ_14','READ_09','READ_08','READ_07'],
'val' :[5,6,7,11,5,7,16,12,13,56,32,13,45,43,46],
})

N=2  # dividing into two dataframes for parallel processing.
dfs = [x for _,x in df.groupby(pd.factorize(df['subject_id'])[0] // N)]

Please find dummy huge dataframe for testing (only the column names match but the data inside is random)

df_size = int(3e7)
N = 30000000
s_arr = pd.util.testing.rands_array(10, N)
df = pd.DataFrame(dict(subject_id=np.random.randint(1, 1000, df_size),
                   readings = s_arr,
                   val=np.random.rand(df_size)
                   ))

Code (with the help of SO users)

import multiprocessing as mp

def transpose_ope(df):                      #this function does the transformation like I want
    df_op = (df.groupby(['subject_id','readings'])['val']
           .describe()
           .unstack()
           .swaplevel(0,1,axis=1)
           .reindex(df['readings'].unique(), axis=1, level=0))
    df_op.columns = df_op.columns.map('_'.join)
    df_op = df_op.reset_index()
    return df_op

def main():

    with mp.Pool(mp.cpu_count()) as pool:
       res = pool.map(transpose_ope, [df for df in dfs])

if __name__=='__main__':
    main()

What transpose_ope does?

It works like this. A subject can have n number of readings. Instead of having his n readings as row, I would like to create the summary statistics of each unique reading as a column. So, subject_id = 1 has 4 readings in total (but 3 unique readings). So, instead of having 4 rows for 1 subject, I would like create features/columns for the readings. But again, instead of readings, I would like to have summary statistics of the readings (MIN,MAX,STDDEV,COUNT,MEAN) etc. So each reading will have 5 columns. So subject_id = 1 will have 15 column in total

Though this code works fine on the sample dataframe, in real life my data has more than four million rows and 30 columns. Moreover, when I apply the transformation of my interest as shown in the code, the column count can go up to 955,500. I'm basically trying to reduce the row count (but increasing the column count).

Can you help?


texts:
I was trying to implement a dp + graph problem. After completing the solution I tried to compare it with solution in tutorial to figure out which one was better and to find a way to improve mine if I could. But I was not able to figure out which one is better, so please help. And if there is a way to improve my algorithm please tell me how I can do so.
Solution in tutorial

Author's solution uses dynamic programming. Let dpi, j be the minimum time required to arrive at the vertex i, if we visit j vertices (including vertices 1 and i). We have a DAG (directed acyclic graph), so we can compute it recursively (and memory constraints were a bit strict in this problem, so it's better to use recursion to compute it). Let's store the transposed version of the graph: if we had an edge (u, v) in the input, we will store (v, u). Then our function calc(i, j), which will compute the answer for dpi, j, will be like that: the base of dynamic programming is dp1, 1 = 0, all other states are equal to «-1». If we call calc(i, j), then it will work like that: if the state we want to compute is incorrect (j < 0), we return a very large integer number (any number that is greater than 109, because T ≤ 109). If the answer for this state has already been calculated, then we return dpi, j (it is easy do determine: if dpi, j ≠  - 1, then it has already been calculated). Else we begin to calculate the state. Firstly, let's put INF (a number greater than 109) into dpi, j. Then look at all the edges beginning in i and try to update dpi, j with the value of calc(to, j - 1) + w (to is the vertex at the endpoint of current edge, w is the weight of this edge). If this value is less than dpi, j, then we update dpi, j and store the information that our last update in dpi, j was from the vertex to. If we try to go by path which doesn't end in the vertex 1, then we get a value which is greater than 109, that's because that the only value we didn't denote as  - 1 is dp1, 1. So, now we have our calc function, let's compute the answer. We will iterate on the number of vertices in the path from n to 1 in descending order, and if calc(n, i) ≤ T, then we have found the answer, now we iterate on the parent vertices we stored while calculating our dp, until we come to vertex 1 (it's important because some participants sent solutions that continued even past vertex 1!) and print the answer.

My implementation of problem
#include<iostream>
using namespace std;

#define INT_MAX 5000

int getAns(int,int);
int n,a[INT_MAX][INT_MAX],dp[2][INT_MAX];
int main(){
    int m,t,u,v,time;
    cin>>n>>m>>t;
    for(int i=0;i<m;i++){
        cin>>u>>v>>time;
        a[u][v]=time;
    }
    cout<<1+getAns(t,1)<<endl;
    int i=1;
    while(dp[1][i]!=0){
        cout<<i<<" ";
        i=dp[1][i];
    }
    cout<<i;
}

int getAns(int time,int curr){
    if(time<0)
        return 0;

    for(int i=1;i<=n;i++){
        if(a[curr][i]!=0&&a[curr][i]<=time){
            int x=1+getAns(time-a[curr][i],i);
            if(dp[0][curr]<x){
                dp[0][curr]=x;
                dp[1][curr]=i;
            }
        }
    }
    return dp[0][curr];
}


texts: Improving a JavaScript filtering function

texts:
I created a function that compares an array of letters to the alphabet, and returns the letters that are not in the original array.

While this works, it looks somewhat clumsy to me, but I don't know what I'm looking for to improve the code...Is there a way to make it more elegant? Perhaps getting rid of the notPresent array somehow?

getLettersNotInContent(['z', 'a', 'p']);

function getLettersNotInContent(letters) {
  // Filter dups if present
  var uniques = [];
  letters.filter(function(letter) {
    if (uniques.indexOf(letter) === -1) {
      uniques.push(letter);
    }
  });

  // Filter and return the letters that are not present in the content
  var alphabet = ['a','b','c','d','e','f','g','h','i','j','k','l','m','n','o',
  'p','q','r','s','t','u','v','w','x','y','z'];
  var notPresent = [];
  alphabet.filter(function(letter) {
    if (uniques.indexOf(letter) === -1) notPresent.push(letter);
  });
  return notPresent;
}


texts:
Recent JavaScript student here. I ran afoul of some weirdness with event listeners that led to my code being unresponsive. After some reading and tweaking, I arrived at a version that worked in JSFiddle, but not in the browser. I finally got it working in the browser, but now my code looks like this:

window.onload = function () {
    function bgnAddChllng() {
        alert("This is a test alert.");
    }
var addChllng = document.getElementById("addChllng");
addChllng.addEventListener('click', bgnAddChllng, false);
}

But the use of window.onload and nesting the actual useful code in an anonymous function seems bloated/hacky to me. What, if anything, can I do to clean it up?

Available on a JSFiddle here.


texts: JSON backup to backend server if connection issues

texts:
I have a program that includes sending data to our backend servers upon completion.

Due to the occasional connection issue at either side I have coded into the main script a JSON dump as a backup in case we are unable to connect to the BE server.

The JSON file is simply a Key/Value mirror of the SQL database fields in the table in question.

I will run this program in the background which will periodically check the folder. It will open the JSON and insert / update any files that are in there and remove them if completed.

from json import load
from time import sleep
from os.path import join
from os import remove
from glob import glob

from sql_tunnel_control import (
    SqlTunnelConnect,
    SSH_KEY,
    sql_ip,
    sql_hostname,
    sql_username,
    sql_password,
    sql_main_database,
    sql_port,
    ssh_host,
    ssh_port,
    ssh_user,
    ssh_password,
    table
)

production = SqlTunnelConnect(
    SSH_KEY,
    None,
    sql_hostname,
    sql_username,
    sql_password,
    table,
    sql_port,
    ssh_host,
    ssh_port,
    ssh_user,
    None,
)

def load_be_data(f):
    """ Save data to json file and allow another program to iterate over that folder until all data is safely in the backend server"""
    try:
        with open(join(f), "r") as j:
            return load(j)
    except Exception as e:
        print(e)
        # pass

def get_files():
    f = glob("backend_data/*.json")
    return f

def send_to_be(f):
    """ takes in a file and attempts to send  to the server.. Returns False if it fails.. Deletes the file if it succeeds """
    print(f)

    if f["query_type"] == "insert":
        create_new_component = """INSERT INTO components .... """
        try:
            production.run_query(create_new_component)
            return True
        except:
            return False
    elif f["query_type"] == "update":
        update_component = """UPDATE components SET ..."""
        try:
            production.run_query(update_component)
            return True
        except:
            return False
    else:
        return False  # problem with query type

def persist_dump():
    to_send = get_files()
    while len(to_send) > 0:
        for f in to_send:
            r = send_to_be(load_be_data(f))
            if r:
                remove(f)
                sleep(1)
            else:
                sleep(300)  # 5 minutes rest.
    # all files completed.. sleep for a bit.. then see whats what//
    sleep(1800)
    persist_dump()

persist_dump()


texts:
I'm trying to write my own stack implementation using LinkedList. It looks like it is doing what it should doing. But how does it look from a design perspective?

If I will go for job interview and show such things to a interviewer, will they say: "that's looks nice" or they rather will look at me with weird look asking "who let you in"?

import java.util.LinkedList;

public class Stack <T> {
private T t;
private LinkedList<T> stack;

public Stack() {
    this.stack = new LinkedList<>();
}

public void push(T t) {
    stack.add(t);
}

public boolean isEmpty(){
    if(stack.size()!=0){
        return false;
    } return true;
}

public T pop(){
    if(!isEmpty()) {
        return stack.removeLast();
    }
    return t;
}

public T peek(){
    if(!isEmpty()){
        return stack.getLast();
    }
    return t;
}

}


texts:
This is my first C program. I want to optimize this simulation.
Algorithm
The simulation algorithm is:

The system can go from \$i\$ to \$i+1\$ with probability \$e^{-L\theta(\rho_i)}\$, where \$\rho_i=i/L\$, and \$\theta(\rho)=\rho(2b-\rho)\$.
When system reaches \$n=0\$, it is revieved (resurected) to position based on how much time it spent on \$n>0\$.
At the end we are intrested in knowing \$\langle\rho\rangle=\sum_{t}\rho_t\$.

Code
Following is the code. I believe this code can be compactified also. I think I do not understand the norms of the ANSI C standard. Feel free to correct me anywhere. I also do not understand if I am properly generating random numbers or not!.
#include<stdio.h>
#include<stdlib.h>
#include<math.h>
#include<time.h>

#define LATTICE 10
#define SWEEPS 100000000
#define BETA 0.5

float rho[LATTICE];

/*
Following function will make variable defined above:
    rho = 1/10, 2/10, 3/10, ... 1.
*/
void initialize()
{
    for(int i = 0; i < LATTICE; i++)
        rho[i] = (i+1.0)/LATTICE;
    return;
}

/*
These are the rates for going form n to n+1 for a given b.
*/
void rates(double *death_rate, float b)
{
    double theta;
    for(int i=0; i < LATTICE; i++)
    {
        theta = rho[i]*(2*b - rho[i]);
        *death_rate = exp(-LATTICE*theta);
        death_rate++;
    }
    return;
}

/*
Following function generates uniform random number
between 0 to 1.
*/
double uniform_rand()
{
    return (double)rand()/(double)RAND_MAX;
}

/*
The following function revive the system when n becomes -1
with the weights = distribution.
*/
int revive(unsigned long long *distribution, unsigned long long norm)
{
    int n = -1;
    double cumsum = 0.0, u_rand = uniform_rand();
    while(cumsum <= u_rand)
    {
        cumsum += (double) *distribution/(double)norm;
        distribution++;
        n++;
    }
    return n;
}

/*
Following function calculate the average density.
*/
double rho_avg(unsigned long long *distribution, unsigned long long norm)
{
    int i;
    double avg_density = 0.0;
    for (i=0; i<LATTICE; i++)
    {
        avg_density += (rho[i]*(*distribution))/norm;
        distribution++;
    }
    return avg_density;
}

double monte_carlo(float b, int n)
{
    unsigned long long i;
    unsigned long long distribution[LATTICE]={0};
    double death_rate[LATTICE];
    srand(time(0));
    rates(death_rate, b);
    for(i = 1; i <= SWEEPS; i++)
    {
        distribution[n] += 1;
        if (uniform_rand() < death_rate[n])
        {
            n--;
            if (n == -1)
                n = revive(distribution, i);
        }
    }
    return rho_avg(distribution, SWEEPS);
}

int main(void)
{
    int i;
    double avg_density;
    initialize();
    avg_density = monte_carlo(BETA, LATTICE-1);
    printf("\nAverage density is %lf\n\n", avg_density);
    return 0;
}


texts: Making currency rate requests in Java with CompletableFuture

texts:
here is my code to get quotes for forex pairs, using the CurrencyLayer API, with CompletableFuture. We do an HTTP GET, parse the JSON response and map it into my data structures (see sample output at the bottom)

Does this look OK?
I'm worried about thread-safety !! :x

Currency.java:

public enum Currency {
    GBP("GBP", "Pound Sterling"),
    EUR("EUR", "Euros"),
    USD("USD", "US Dollars");

    public final String symbol;
    public final String name;
    Currency(String symbol, String name) {
        this.symbol = symbol;
        this.name = name;
    }
}

CurrencyPair.java:

public class CurrencyPair {
    final public Currency base;
    final public Currency counter;

    public CurrencyPair(Currency base, Currency counter) {
        this.base = base;
        this.counter = counter;
    }
}

CurrencyLayer.java:

import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.CompletableFuture;
import javax.ws.rs.client.Client;
import javax.ws.rs.client.ClientBuilder;
import javax.ws.rs.core.Response;

import com.google.gson.JsonElement;
import com.google.gson.JsonObject;
import com.google.gson.JsonParser;
import models.Currency;

public class CurrencyLayer implements ForexQueryService {
    private final Client client = ClientBuilder.newClient();

    private CompletableFuture<Response> makeRequest(List<CurrencyPair> pairs) {
        final String url = "http://www.apilayer.net/api/live?access_key=[my key ^^]&format=1";
        return CompletableFuture.supplyAsync(() -> client.target(url).request().get());
    }

    private static double getUSDto(JsonObject quotes, Currency c) {
        if (c == Currency.USD) {
            return 1.0;
        } else {
            return quotes.get("USD" + c.symbol.replace("XBT", "BTC")).getAsDouble();
        }
    }

    private static List<ForexRate> convertResponse(Response r, List<CurrencyPair> pairs) {
        final String body = r.readEntity(String.class);

        // log this data
        System.out.println(body);

        JsonElement jelement = new JsonParser().parse(body);
        JsonObject jo = jelement.getAsJsonObject();
        JsonObject quotes = jo.get("quotes").getAsJsonObject();

        List<ForexRate> ret = new ArrayList<>();
        for (CurrencyPair p : pairs) {
            ret.add(new ForexRate(p.base, p.counter, getUSDto(quotes,p.counter)/getUSDto(quotes, p.base)));
        }

        return ret;
    }

    public CompletableFuture<List<ForexRate>> lookUpRates(List<CurrencyPair> pairs) {
        return makeRequest(pairs).thenApply((t) -> convertResponse(t, pairs));
    }
}

ForexRate.java

public class ForexRate {
    public final Currency from;
    public final Currency to;
    public final Double rate;

    public ForexRate(Currency from, Currency to, Double rate) {
        this.from = from;
        this.to = to;
        this.rate = rate;
    }
}

Main.java:

public class Main {
    static public void main(String[] args) {
        CurrencyLayer c = new CurrencyLayer();
        try {
            List<CurrencyPair> pairs = new ArrayList<>();
            pairs.add(new CurrencyPair(Currency.GBP,Currency.USD));
            pairs.add(new CurrencyPair(Currency.JPY,Currency.GBP));

            List<ForexRate> rates = c.lookUpRates(pairs).get();
            for (ForexRate rate : rates) {
                System.out.println(rate.from + " " + rate.to + " -> " + rate.rate);
            }

        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

Sample output:

{
  "success":true,
  "terms":"https:\/\/currencylayer.com\/terms",
  "privacy":"https:\/\/currencylayer.com\/privacy",
  "timestamp":1505384954,
  "source":"USD",
  "quotes":{
    "USDAUD":1.249706,
    "USDEUR":0.840197,
    "USDGBP":0.75723
  }
}
GBP USD -> 1.320602723082815
EUR GBP -> 0.9012529204460382


texts: Storing localStorage data as part of rxjs reducer?

texts:
I use ngrx in my Angular app to state management. At a certain time I was asked to save some variable to local storage and retrieve it with first load.
That made sense to make it as part of the effects, instead of managing one logic for localStorage and another logic for the selectors. So this is what I did;
I extended ngrx Action class as following:
import { Action } from '@ngrx/store';

export class StorableAction implements Action {
    type: string;
    payload?: any;
    localStorageKey?: string;
}

Implemented it in the actions (snapshots.actions.ts):
export class ChangeResolutionWidth implements StorableAction {
    public readonly type = ESnapshotsActionType.ChangeResolutionWidth;
    public readonly localStorageKey = LocalStorageKey.SnapshotsResolutionWidth;

    constructor(public payload: number) { }
}

Then in snapshots.effects.ts:
constructor(
    private channelsService: ChannelsService,
    private actions$: Actions,
    private store: Store<SnapshotsState>
) {
  this.actions$.subscribe(
    (action: StorableAction) => {
      if (action.localStorageKey && action.payload){
        localStorage.setItem(action.localStorageKey, JSON.stringify(action.payload));
      }
    })
  }

Now, whenever I load a component, I do this.store.select with take(1) to selectResolutionWidth (in the example above). If the value is undefined, then I take it from the localStorage and dispatch it. That felt like a sweet solution - whenever I needed to store some state value to the local storage, all I needed is to use StorableAction instead of Action and name the key.
My questions are:

would you say it is a proper implementation of store effects, or am I bending the separation of concerns principal?
Now I've come to a scenario in which I need to store a value to the local storage that is actually not needed in the state (I only use it in one component). That brings up a conflict - should I use the same mechanism I created, or store to the local storage directly (which will cause a duplication, when localStorage is being managed both in state level and component level). What's better?

Thanks!


texts:
This is a question from the book "Cracking the Coding Interview".

  Write code to reverse a C-Style String (C-String means that “abcd” is
  represented as five characters, including the null character )

Please review this code.

#include <iostream>

void inPlaceReverseString(char str[], int size)
{
    int j = size - 2;
    int i;

    for (i = 0; i < (size-1)/2; ++i)
    {
        std::swap(str[i], str[j]);
        j--;
    }
    str[size - 1] = '\0';
}

int main()
{
    char str[255];
    std::cout << "Enter String\n";
    std::cin.getline(str, 255);
    std::cout << "String: " << str << '\n';
    int size = 0;
    while (str[size] != '\0')
    {
        size++;
    }
    size++;
    inPlaceReverseString(str, size);
    std::cout << "Reversed String: " << str << '\n';
}


texts: Efficient visualization of large number of particles using openGL

texts:
I have been looking into using some software to perform industrial-scale DEM simulations. These type of simulations will require millions of particles to be simulated.

Currently, the software I would like to use has some notable performance issues when visualizing the results during the simulation when using openGL. It turns out the part of the code responsible for drawing the particles is this:

void Draw_ParticleGeometry_Sphere()
{
  SetColor_Ramp();

//if(m_OpenGL.Debug_Lines==1)
//{
//  Gl_WireframeP = true;
//}

  for( int i=0; i<m_num_KParticleObjects; i++ )
  {
     glNewList(10+i,GL_COMPILE);

    if(m_OpenGL.color_type==0)
    {
     glMaterialfv(GL_FRONT, GL_AMBIENT_AND_DIFFUSE, Col[i]);
     glMaterialfv(GL_FRONT, GL_SPECULAR, materialSpecular);
     glMaterialfv(GL_FRONT, GL_EMISSION, materialEmission);
     glMaterialf(GL_FRONT, GL_SHININESS, shininess);
    }
          glPushMatrix();

          glutSolidSphere(m_KParticleObject[i].radius,20,20);

          if(Gl_WireframeP)
          {

                 glMaterialfv(GL_FRONT, GL_AMBIENT_AND_DIFFUSE, Col[6]);
                 glMaterialfv(GL_FRONT, GL_SPECULAR, materialSpecular);
                 glMaterialfv(GL_FRONT, GL_EMISSION, materialEmission);
                 glMaterialf(GL_FRONT, GL_SHININESS, shininess);
            glutWireSphere(m_KParticleObject[i].radius*1.02f,15,15);
          }
          glPopMatrix();

     glEndList();

  }

}

To be fair I'm new to openGL, but it seems to me that this piece of code loops over all particles, assigns them attributes and add them to a list. Generally, looping like this is not very efficient for a large number of elements. I have been reading up in visualizing particles in openGL and it seems instancing would be a better fit. My questions are:

Is looping over all particles like this an efficient way of handling such large systems of particles in openGL?
Is there an alternative to looping over all particles?
How could this code be improved?


texts:
I have the following code for separating out overlapping rectangles vs.
non-overlapping rectangles in C++ 11:

#include <stdio.h>
#include <opencv/cv.h>
#include <opencv/highgui.h>
#include <opencv/cxcore.h>
#include <math.h>
#include <array>

int main() {
    auto detected = get_detected();
    std::vector <Rect> singles;
    std::vector <Rect> overlaps;
    for (auto rect_1: detected) {
        bool isSingle = false;
        for (auto rect_2: detected) {
            if (rect_1 == rect_2) {
                continue;
            }
            bool intersects = ((rect_1 & rect_2).area() > 0);
            if (intersects) {
                isSingle = false;
                break;
            }
        }
        if (isSingle) {
            singles.push_back(rect_1);
        } else {
            overlaps.push_back(rect_1);
        }
    }

}

How do I make this cleaner? I'm thinking of doing singles.push_back(std::move(rect_1));, but I'm not sure if that's a good idea.

I would love some guidance in the right direction.


texts: Check if a JavaScript variable exists and checking its value

texts:
I feel like this bit of code could be condensed or made better in some way. It may in fact not be, but figured I'd get some people to have a look. I have multiple pages on my site and in certain pages I'm setting a JavaScript variable:

var header_check = "user-profile";

Some pages I'm not setting that variable. In another JavaScript file that gets loaded on the page I check if that variable exists and perform various actions if it does.

Is this the best way to check if the variable exists? Also, is this the best way to see if user-profile is set?

var header_cookie = typeof header_check !== 'undefined' ? 'user-profile' : 'admin-profile';
var cookie_check = header_cookie == 'user-profile' ? true : false;

var city_profile = cookie_check ? 'userCity' : 'city';
var state_profile = cookie_check ? 'userState' : 'state';


texts:
I am having a problem with my program: it does as required but largest_blanagram() is too slow. I'm not sure whether the best way to speed it up is (1) try to speed up blanagram(), which is being called a lot (2) figure out a way to call blanagram() less. I'm not sure how to go about either of these things particularly.

Things I've been thinking: is there a way to use another dictionary more efficiently, such as using map()?

# CW Q2 blanagram.py

import string
from Q1_anagram import readfile, make_anagram_dict # change

def blanagram(word, anagramdict): # split up into smaller functions?
   """
   """
    word = sorted(word) # sort alphabetically
    alphabet = list(string.ascii_lowercase)
    index = 0
    blanagrams = []
    for char in word:
        original_char = char # save original char, so can revert back to it later
        for letter in alphabet:
            word[index] = letter
            word_copy = word # as order messes up when order alphabetically, so save current state and revert back to it after the iteration
            word = ''.join(sorted(word)) # list -> string to be able to check dict and sort alphabetically
            if word in anagramdict.keys() and anagramdict[word] not in blanagrams:
                for anagramdict[word] in anagramdict[word]:
                    blanagrams.append(anagramdict[word])# change name?
            word = word_copy
        # next character in word
        word = list(word) # string -> list in order to index
        word[index] = original_char # set back to original char, as only 1 char can be different from original word
        index += 1 # to change following char

    return blanagrams # get rid of anagrams?

def largest_blanagram(words, word_length): # greatest number of blanagram variants
    """
    """
    blanagrams = {}
    length = 0
    for word in words:
        if len(word) == word_length:
            variants = blanagram(word, make_anagram_dict(words))
            #print(variants)
            if len(variants) > length:
                blanagrams[word] = variants
                length = len(variants)

    return blanagrams

if __name__ == "__main__":
    # print(blanagram('ghost', make_anagram_dict(readfile('words.txt'))))
    import sys
    print(largest_blanagram(readfile('words.txt'), 13)) #sys.argv[1]

Imports -

def readfile(filename):
"""
"""
file = open(filename,'r').readlines() # look into WITH open file
words = []
for line in file:
    line = line.lower().split() # orang and Orang? unique? or is asteer another answer
    for word in line:
        words.append(word)
return words

def make_anagram_dict(words):
"""
"""
assert len(words) != 0
anagram_dict = {}
for word in words:
    alpha = ''.join(sorted(word))  # Order word alphabetically
    # Key is alphabetically ordered word, value is list of anagrams of the key
    anagram_dict.setdefault(alpha, []).append(word)  # So not appending NoneType (as list.append returns None)

return anagram_dict

And here is a pastebin equivalent (pastebin has file size limit) link to the words.txt file that I use.


texts:
I want to implement a Logger and use Serilog as Provider. My Project is ASP.NET MVC Framework (not core).

Initially I tried to replicate ASP.Net Core Logging framework. I created a LogConfig static class, and initialized it from Global.asax.

public static class LogConfig {
  public static LoggerFactory LoggerFactory = new LoggerFactory();
  public static void RegisterLogger() {
  LoggerFactory = new LoggerFactory();
  Logger = new LoggerConfiguration().ReadFrom.AppSettings().CreateLogger();
  LoggerFactory.AddSerilog();
  }

   public static ILogger<T> CreateLogger<T>() {
        return LoggerFactory.CreateLogger<T>();
    }
}

Initially I wanted to inject the logger into controller, using ninject, but could not figure out how to do this.

For example, something like this:

private static void RegisterServices(IKernel kernel) {
kernel.Bind<typeof(ILogger<>).ToProvider(LogConfig.LoggerFactory.CreateLogger<>());
}

But it did not work and I found it very complicated. Then, I decided to just use my logger factory and initialize it the controller like this:

ILogger<MyController> logger = LogConfig.CreateLogger<MyCotroller>();

But then, I wanted to access this logger in my ExceptionLoggingFilter, but again it started to get complicated. So in the end I got rid of Microsoft LoggerFactory and just used Serilog.

This is my final code:

Logconfig.cs, which is in App_start and is called from Global.asax:

public static class LogConfig
{
    public static ILogger Logger { get; set; }

    public static void RegisterLogger()
    {
        Logger = new LoggerConfiguration().ReadFrom.AppSettings().CreateLogger();
    }
 }

ExceptionLoggingFilter.cs, which is in the Filters folder:

public class ExceptionLoggingFilter : FilterAttribute, IExceptionFilter
{
    public void OnException(ExceptionContext filterContext)
    {
        var controllerName = filterContext.RouteData.Values["controller"];
        var actionName = filterContext.RouteData.Values["action"];

        // send ajax response
        if (string.Equals(filterContext.HttpContext.Request.Headers["X-Requested-With"], "XMLHttpRequest", StringComparison.InvariantCultureIgnoreCase))
        {
            filterContext.Result = new JsonResult
            {
                JsonRequestBehavior = JsonRequestBehavior.AllowGet,
                Data = new
                {
                    Message = "An error has occured, please try again later.",
                }
            };
        }

        LogConfig.Logger.Error(filterContext.Exception, $"Exception occured in Controller: {controllerName}, Action: {actionName}.", filterContext.Controller.GetType());
        filterContext.HttpContext.Response.StatusCode = 500;
        filterContext.ExceptionHandled = true;
    }
}

So, finally, there is no injection, and no logger factory, only a static global logger which can be accessed everywhere. Is this a good solution?


texts: Getting the grandparent directory of the current code

texts:
I have this example:

from os.path import dirname, realpath
dirname(dirname(dirname(realpath(__file__)))

And the chained dirname() applications look a bit odd to me. Is there a way to improve my code? There is of course something like Better Function Composition in Python, but that seems too complicated to me.

Maybe, some solution with function composition "∘" would be better?

reduce(∘, [dirname] * 3)(realpath(__file__))

But that doesn't look any better.

Do you have any ideas for this concrete example?